Interacting with hierarchical clusters of video segments using a metadata search

ABSTRACT

Embodiments are directed to techniques for interacting with a hierarchical video segmentation by performing a metadata search. Generally, various types of metadata can be extracted from a video, such as a transcript of audio, keywords from the transcript, content or action tags visually extracted from video frames, and log event tags extracted from an associated temporal log. The extracted metadata is segmented into metadata segments and associated with corresponding video segments defined by a hierarchical video segmentation. As such, a metadata search can be performed to identify matching metadata segments and corresponding matching video segments defined by a particular level of the hierarchical segmentation. Matching metadata segments are emphasized in a composite list of the extracted metadata, and matching video segments are emphasized on the video timeline. Navigating to a different level of the hierarchy transforms the search results into corresponding coarser or finer segments defined by the level.

BACKGROUND

Recent years have seen a proliferation in the use of video, which hasapplications in practically every industry from film and television toadvertising and social media. Businesses and individuals routinelycreate and share video content in a variety of contexts, such aspresentations, tutorials, commentary, news and sports segments, blogs,product reviews, testimonials, comedy, dance, music, movies, and videogames, to name a few examples. Video can be captured using a camera,generated using animation or rendering tools, edited with various typesof video editing software, and shared through a variety of outlets.Indeed, recent advancements in digital cameras, smartphones, socialmedia, and other technologies have provided a number of new ways thatmake it easier for even novices to capture and share video. With thesenew ways to capture and share video comes an increasing demand for videoediting features.

Conventionally, video editing involves selecting video frames andperforming some type of action on the frames or associated audio. Somecommon operations include importing, trimming, cropping, rearranging,applying transitions and effects, adjusting color, adding titles andgraphics, exporting, and others. Video editing software, such as ADOBE®PREMIERE® PRO and ADOBE PREMIERE ELEMENTS, typically includes agraphical user interface (GUI) that presents a video timeline thatrepresents the video frames in the video and allows the user to selectparticular frames and the operations to perform on the frames. However,conventional video editing can be tedious, challenging, and even beyondthe skill level of many users.

SUMMARY

Embodiments of the present invention are directed to segmentation andhierarchical clustering of video. In an example implementation, a videois ingested to generate a multi-level hierarchical segmentation of thevideo. In some embodiments, the finest level of the hierarchy consistsof or otherwise identifies a smallest interaction unit of thevideo—semantically defined video segments of unequal duration calledclip atoms. Clip atom boundaries are detected in various ways. Forexample, speech boundaries are detected from audio of the video, sceneboundaries are detected from video frames of the video, and eventboundaries are detected from a temporal log associated with the video(e.g., a software usage log generated while screen capturing orscreencasting). The detected boundaries are used to define the clipatoms, which are hierarchically clustered to form a multi-levelhierarchical representation of the video. In some cases, thehierarchical segmentation identifies a static, pre-computed,hierarchical set of video segments, where each level of the hierarchicalsegmentation identifies a complete set (i.e., covering the entire rangeof the video) of disjoint (i.e., non-overlapping) video segments with acorresponding amount of granularity. Hierarchical video segmentationenables new ways to create, edit, and consume video.

For example, some embodiments are directed to techniques for interactingwith a hierarchical video segmentation using a video timeline. Apresented video timeline can be segmented into selectable video segmentsdefined by one of the levels of the hierarchical segmentation, and oneor more video segments can be selected through interactions with thevideo timeline. For example, a click or tap on a video segment or a dragoperation dragging along the timeline snaps selection boundaries tocorresponding segment boundaries defined by the level. Navigating to adifferent level of the hierarchy transforms the selection into coarseror finer video segments defined by the level, enabling a refinedselection of a desired portion of a video. Any operation can beperformed on selected video segments, including playing back, trimming,or editing.

Some embodiments are directed to techniques for interacting with ahierarchical video segmentation using a metadata panel presenting acomposite list of video metadata. Generally, various types of metadatacan be extracted from a video, such as a transcript of audio of thevideo, keywords from the transcript, content or action tags visuallyextracted from video frames, and action tags extracted from anassociated temporal log. A composite list of the extracted metadata canbe segmented into selectable metadata segments at locationscorresponding to boundaries of video segments defined by a particularlevel of the hierarchical segmentation. One or more metadata segmentscan be selected from the metadata panel in various ways, such as byclicking or tapping on a metadata segment, or an interaction elementassociated with the metadata segment. When a metadata segment isselected, a corresponding video segment is emphasized on the videotimeline, a playback cursor is moved to the first video frame of thevideo segment, and the first video frame is presented. Navigating to adifferent level of the hierarchy updates the composite list into coarseror finer metadata segments corresponding to the boundaries defined bythe level, enabling a refined selection of a desired portion of theextracted metadata and a corresponding portion of the video.

Some embodiments are directed to techniques for interacting with ahierarchical video segmentation by performing a metadata search.Generally, extracted metadata can be segmented into metadata segmentsand associated with corresponding video segments defined by ahierarchical video segmentation. As such, a metadata search can beperformed to identify matching metadata segments and correspondingmatching video segments defined by a particular level of thehierarchical segmentation. Matching metadata segments are emphasized ina composite list of the extracted metadata, and matching video segmentsare emphasized on the video timeline. Navigating to a different level ofthe hierarchy transforms the search results into corresponding coarseror finer segments defined by the level, enabling a refined selection ofa desired portion of the video.

As such, hierarchical video segmentation enables new ways to create,edit, and consume video, and gives creators and consumers a moreintuitive structure for interacting with video.

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is described in detail below with reference to theattached drawing figures, wherein:

FIG. 1 is a block diagram of an example computing system for videoediting or playback, in accordance with embodiments of the presentinvention;

FIGS. 2A-2B are illustrations of example boundary adjustments for videosegments based on voice activity detection, in accordance withembodiments of the present invention;

FIG. 3 is an illustration of an example hierarchical segmentation of avideo, in accordance with embodiments of the present invention;

FIG. 4 is an illustration of an example user interface for interactingwith hierarchical clusters of video segments, in accordance withembodiments of the present invention;

FIGS. 5A-5I are illustrations of example interactions with hierarchicalclusters of video segments using a video timeline window and/or azoom/scroll bar, in accordance with embodiments of the presentinvention;

FIGS. 6A-6J are illustrations of example user interfaces for interactingwith hierarchical clusters of video segments using a metadata paneland/or a metadata search, in accordance with embodiments of the presentinvention;

FIG. 7 is an illustration of an example user interface for interactingwith hierarchical clusters of video segments based on software logevents, in accordance with embodiments of the present invention;

FIG. 8 is a flow diagram showing a method for generating a hierarchicalsegmentation of a video timeline, in accordance with embodiments of thepresent invention;

FIG. 9 is a flow diagram showing a method for hierarchically clusteringsemantic video segments into a hierarchical segmentation, in accordancewith embodiments of the present invention;

FIG. 10 is a flow diagram showing a method for detecting boundaries ofclip atoms, in accordance with embodiments of the present invention;

FIG. 11 is a flow diagram showing a method for detecting and adjustinglocations of speech boundaries, in accordance with embodiments of thepresent invention;

FIG. 12 is a flow diagram showing a method for snapping speechboundaries to proximate scene boundaries, in accordance with embodimentsof the present invention;

FIG. 13 is a flow diagram showing a method for extracting eventboundaries of log events from a temporal log, in accordance withembodiments of the present invention;

FIG. 14 is a flow diagram showing a method for forming different levelsof a hierarchical segmentation, in accordance with embodiments of thepresent invention;

FIG. 15 is a flow diagram showing a method for selecting a video segmentdefined by a hierarchical segmentation, in accordance with embodimentsof the present invention;

FIG. 16 is a flow diagram showing a method for executing an operation onan identified cluster defined by a hierarchical segmentation, inaccordance with embodiments of the present invention;

FIG. 17 is a flow diagram showing a method for updating a presentationof a first level of the hierarchical segmentation in response tonavigating to a different level, in accordance with embodiments of thepresent invention;

FIG. 18 is a flow diagram showing a method for snapping a selection toboundaries of clusters of clip atoms, in accordance with embodiments ofthe present invention;

FIG. 19 is a flow diagram showing a method for snapping a selection toboundaries of clusters semantic video segments, in accordance withembodiments of the present invention;

FIG. 20 is a flow diagram showing a method for selecting video segmentsusing a drag operation, in accordance with embodiments of the presentinvention;

FIG. 21 is a flow diagram showing a method for emphasizing a videosegment in response to an input identifying a selectable metadatasegment, in accordance with embodiments of the present invention;

FIG. 22 is a flow diagram showing a method for updating a video timelinein response to an input identifying a metadata segment, in accordancewith embodiments of the present invention;

FIG. 23 is a flow diagram showing a method for executing a search ofextracted metadata and emphasizing matching video segments on a videotimeline, in accordance with embodiments of the present invention;

FIG. 24 is a flow diagram showing a method for executing a search of theextracted metadata and updating a selection state for matching videosegments, in accordance with embodiments of the present invention;

FIG. 25 is a flow diagram showing a method for updating matching andvideo metadata segments in response to an input navigating to adifferent level of a hierarchical segmentation, in accordance withembodiments of the present invention; and

FIG. 26 is a block diagram of an example computing environment suitablefor use in implementing embodiments of the present invention.

DETAILED DESCRIPTION

Overview

A video file, clip, or project can usually be split up into visual andaudio elements. For example, a video might encode or otherwise identifya video track comprising a sequence of still images (e.g., video frames)and an accompanying audio track comprising one or more audio signals.Conventionally, video editing tools provide an interface that lets usersperform time-based editing on selected video frames. In other words,conventional video editing generally involves representing a video as asequence of fixed units of equal duration (e.g., video frames) andpresenting a video timeline that allows the user to select and interactwith particular video frames. However, interaction modalities that relyon a selection of particular video frames or a corresponding time rangeare inherently slow and fine-grained, resulting in editing workflowsthat are often considered tedious, challenging, or even beyond the skilllevel of many users. In other words, time-based video editing thatrequires selecting particular video frames or time ranges provides aninteraction modality with a fixed granularity, resulting in aninflexible and inefficient interface. As such, there is a need for animproved interface and improved interaction modalities for video editingtools.

Accordingly, embodiments of the present invention are directed tosegmentation and hierarchical clustering of video, and variousinteraction modalities for video editing and playback based onhierarchical clusters of video segments. In an example implementation, avideo is ingested to generate a multi-level hierarchical segmentation ofthe video. In some cases, the hierarchical segmentation identifies astatic, pre-computed, hierarchical set of video segments, where eachlevel of the hierarchical segmentation includes or otherwise identifiesa complete set (i.e., covering the entire range of the video) ofdisjoint (i.e., non-overlapping) video segments. In some embodiments,the finest level of the hierarchy consists of or otherwise identifies asmallest interaction unit of the video—semantically defined videosegments of unequal duration called clip atoms, and the clip atoms arehierarchically clustered to form a multi-level hierarchicalrepresentation of the video. Hierarchical video segmentation enables newways to create, edit, and consume video. As explained in more detailbelow, it gives creators and consumers a more intuitive structure forinteracting with video.

More specifically, hierarchical segmentation gives creators andconsumers a new interaction modality that can be used to browse, edit,and playback videos. Depending on the implementation, a video ishierarchically segmented into hierarchical clusters of video segments(e.g., clip atoms), where the boundary locations for the video segmentsare determined based on the content of the video (e.g., the presence ofspeech, scene transitions, associated software events such as softwaretool events depicted in the video). Thus, in some embodiments,boundaries for video segments are placed at semantically meaningfulparts of the video, and the hierarchical segmentation clusters theresulting video segments with multiple levels of granularity.

To interact with these hierarchical clusters, one or more interactionelements (e.g., a video timeline, zoom bar, scroll bar, metadata panel,search bar, clip detail tool for changing an active hierarchy level,and/or others) allow users to make a refined selection of video segmentsthat snaps to semantically meaningful portions of the video with adesired level of granularity. For example, rather than simply providinga video timeline segmented by some fixed unit of equal duration (e.g., aframe, a second) in a manner that is divorced from semantic meaning,interactions with hierarchical clusters of semantic video segmentsprovide a more flexible and efficient interaction modality and userinterface, allowing users to quickly identify, select, and operate onportions of a video that are likely to be of interest. As such, editorscan now work more quickly and consumers can now jump to the section ofinterest without having to watch the video.

Ingestion, Segmentation, and Hierarchical Clustering of Video

In an example high-level process, a video is ingested by segmenting thevideo into clip atoms, hierarchically clustering the clip atoms to formvideo segments, extracting metadata about the video, and associating theextracted metadata with corresponding video segments.

In some embodiments, a hierarchical segmentation of a video is generatedby computing an over-segmentation of the video's timeline to identifyboundaries for the clip atoms, and the clip atoms are hierarchicallyclustered to form the hierarchical segmentation. In an exampleimplementation, an over-segmentation of the video timeline is computedby applying one or more detection techniques to detect candidateboundaries for clip atoms, applying one or more adjustment techniques toadjust the candidate boundaries to identify the clip atom boundaries,and generating or otherwise storing a representation of the clip atomboundaries and/or the corresponding clip atoms. In some embodiments, theclip atoms form the finest (lowest) level of the hierarchicalsegmentation, and higher levels are formed by merging clusters ofconsecutive clip atoms into larger video segments (corresponding tolarger chunks on the timeline). In some embodiments, one or more cutcosts are computed and assigned (e.g., as metadata) or otherwiseassociated with each boundary, a cost function is defined based on thecut costs for each boundary, and a line breaking, dynamic programming,or other segmentation algorithm is used to compute an optimalsegmentation based on the cost function. The segmentation algorithm isrepeatedly applied to successive levels of the hierarchy, clustering thevideo segments from a particular level into coarser and longer segmentsuntil reaching a desired top level, for example, when the entiretimeline becomes a single chunk, until the number of video segments in ahierarchy level is smaller than some number (e.g., 10), or otherwise.The output is a multi-level hierarchical representation of the video.

In some embodiments, candidate boundaries for video segments (e.g., clipatoms) are detected using one or more detection techniques. In variousembodiments, any number and combination of detection techniques areapplied to identify speech boundaries (e.g., utterance boundaries, wordboundaries, sentence boundaries), scene boundaries, event boundariesderived from software log events, and/or other types of boundaries.

For example, in some embodiments, any known speech-to-text algorithm isapplied to an audio track associated with a video to generate atranscript of speech, detect speech segments (e.g., corresponding towords, sentences, utterances of continuous speech separated by audiogaps, etc.), detect non-speech segments (e.g., pauses, silence, ornon-speech audio), and/or the like. In various embodiments, thetranscript is associated with the video timeline, speech segments of thetranscript are mapped to locations on the video timeline, and locationsof candidate speech boundaries (e.g., utterance boundaries, wordboundaries, sentence boundaries) are identified at the start and end ofcorresponding speech segments. In some cases, the transcript is parsedinto speech segments and/or non-speech segments, for example, byapplying natural language processing based on linguistic features of thetranscript (e.g., using boundary detection logic), using a naturallanguage processing model (e.g., a machine learning model), some otherkind of segmentation technique, and/or other techniques. In someembodiments, to segment the video's timeline into a complete anddisjoint set of speech segments, gaps between speech segments areconsidered to be speech segments (e.g., with a silence label applied)and/or considered to be silence segments (or silence gaps).

In another example embodiment, scene boundaries are detected from videoframes of the video. A scene boundary (also called a shot boundary) is avideo cut or other visible scene transition in the video. In some cases,a video cut is a hard cut between two adjacent video frames, amulti-frame video cut that spans a sequence of multiple video frames(e.g., a fade or a wipe), and/or other types. Depending on theimplementation, a scene boundary can be generated by recording a videowith consecutive takes, by concatenating two different takes (e.g.,using video editing software), by applying a visual transition (e.g.,using video editing software), by switching between multiple cameras,and/or other ways. In some embodiments, scene boundaries are identifiedby detecting abrupt visual changes in video frames using any knowntechnique. In some situations where a detected scene boundary is basedon a video cut between two adjacent video frames, the scene boundary isidentified at a location on the video timeline between the video frames.In some embodiments where a detected scene boundary is based on amulti-frame video cut that spans a sequence of more than two videoframes, the scene boundary is identified at a location on the videotimeline that is centered (or at some other location) in the sequence ofvideo frames.

In another example of possible candidate boundaries for video segments,in some embodiments, one or more usage logs associated with the videoare accessed and used to detect log events and corresponding eventboundaries. Depending on the implementation, various types of log eventsare detected from various types of usage logs. For example, in oneimplementation involving screen captured or screencast videos oftutorials for creative software such as ADOBE PHOTOSHOP® or ADOBEFRESCO®, a software usage log generated by the creative software whilescreen capturing or screencasting is read to identify the times whendetected log events such as tool events (e.g., indicating a selection,change, or use of a particular software tool, such as select brush,create layer, etc.) occurred. In an example gaming implementation, asoftware usage log is read to identify event boundaries for detectedsoftware log events such as leveling up or beating an enemy. In anexample cooking implementation, a usage log is read to identify eventboundaries for visually detected events in the video such as a userpicking up a different pot or cooking tool. In some cases, the log neednot correspond to events derived from video frames. For example, in anexample implementation with a live chat or chat stream associated with alivestreamed video, a corresponding user chat log or session is read toidentify event boundaries such as chat messages about a particulartopic. In an example video streaming implementation (whetherlivestreaming or viewing archived video), a usage log representing how auser(s) has viewed the video is read to identify event boundaries fordetected interaction events such as navigational events (e.g., play,pause, skip). Generally, any type of temporal log and/or metadata can beread to identify event boundaries.

In some cases (e.g., if a log does not report times relative to thevideo timeline), an event timeline represented in the log is converted,mapped, or otherwise associated with the video timeline. As such, insome embodiments, the times of log events captured by a log areextracted (e.g., by reading from structured data fields, by applyingsearch patterns, natural language processing, and/or other rules tounstructured data, etc.), the times of the log events are mapped tolocations on the video timeline (if necessary), and event boundaries(e.g., tool boundaries identifying locations of tool events) areidentified at the corresponding locations on the video timeline.

In some embodiments, detected candidate boundaries are adjusted usingone or more adjustment techniques. In some cases, certain types ofcandidate boundaries are adjusted using certain types of adjustmenttechniques. In various implementations, candidate boundaries derivedfrom an audio track of the video that includes speech (i.e., speechboundaries, such as utterance or sentence boundaries) are adjusted usingvoice-activity-detection (VAD) and/or snapped to proximate sceneboundaries (e.g., when a scene boundary falls within a silence), eventboundaries derived from log events are adjusted (e.g., tool boundariesthat fall within a short silence, such as less than one second, aresnapped to the closest non-tool boundary), and/or other adjustmenttechniques are applied.

In some embodiments, candidate boundaries for video segments areadjusted using VAD. VAD is an audio technique that detects the presence(or likelihood of the presence) of human voice in an audio signal at aparticular time. In some cases, VAD scores are normalized to acontinuous range, for example, on [0, 1] such that VAD=1 means there ishuman voice and VAD=0 means no human voice. As such, in someembodiments, to avoid placing a video segment boundary (e.g., a clipatom boundary) in the middle of speech, VAD score is used as a cut costto adjust the location of certain boundaries (e.g., by adjustingboundaries with high VAD cost and/or permitting boundaries with low VADcost). For example, in some cases, candidate speech boundaries arerefined by snapping the candidate boundaries to locations within aneighborhood of the candidate boundaries where VAD scores are at aminimum. In another example, a gap of silence between two speech chunks(e.g., a silence gap that is shorter than some duration d, such as onesecond) may be closed by searching the silence gap for the lowest VADscore and merging the candidate speech boundaries surrounding thesilence gap into an adjusted boundary at the location of the lowest VADscore.

In some embodiments, smoothing is applied to the VAD scores prior toadjusting candidate speech boundaries. Instead of cutting or otherwisedefining a video segment boundary right at the end of the speech,smoothing the VAD prior to snapping boundaries to local VAD minimaeffectively adds a temporal buffer to speech boundaries, thereby cuttingor otherwise defining boundaries for speech chunks at some distance awayfrom the unsmoothed boundaries, which generates more natural transitionsbetween video segments. In an example embodiment, any known VADtechnique is applied to some or all of an audio track associated withvideo to calculate VAD scores, smoothing is applied (e.g., using akernel such as a Gaussian or Cauchy kernel, by applying a filter, etc.)to the VAD scores, and the locations of candidate speech boundaries areadjusted based on the smoothed VAD scores. In some embodiments,smoothing is only applied to the VAD scores at the location of (e.g.,centered around) candidate speech boundaries. In some cases, the widthof a smoothing kernel or filter corresponds with, or sets a minimum for,the neighborhood to search. Since smoothing a signal generally leaves atail, and the size of the tail generally corresponds to the size of thekernel or filter (e.g., width r), some embodiments set the neighborhoodto search greater than or equal to the size of the kernel, filter, ortail (e.g., for each audio-derived candidate boundary such as eachcandidate speech boundary, search the lowest VAD value in a neighborhoodof width r). These are just a few examples, and other embodiments applyadditional or alternative smoothing techniques.

In some embodiments, candidate speech boundaries are adjusted bysnapping the candidate speech boundaries to proximate scene boundaries.In various embodiments, speech boundaries (e.g., utterance boundaries,sentence boundaries) are determined by analyzing audio associated withthe video (e.g., transcribing speech from the audio and detectingutterance, word, and/or sentence boundaries from the transcript), whilescene boundaries are determined by analyzing video frames of the video.In some cases, there can be slight discrepancies between the two typesof boundaries. If the video is cut (or a boundary placed) just before orafter a scene boundary, it causes a jarring effect (a sudden jump at thebeginning or end of the cut or segment). To avoid such jarring cuts andto account for discrepancies, in some embodiments, the locations of someor all audio-derived candidate boundaries (e.g., speech boundaries) areadjusted. In an example implementation, scene boundaries that fallwithin a silence are detected (e.g., based on looking up and finding noword at a corresponding portion of an associated transcript, based on anassociated VAD score being below some threshold value), and proximatespeech boundaries (which can also be thought of as silence boundariessince these speech boundaries divide speech and silence) are snapped tothe scene boundaries. That is, in some embodiments, silence boundariesthat are proximate to (e.g., separated by less than some thresholdduration such as within 500 milliseconds of) a scene boundary that fallswithin silence are snapped to the scene boundary. In some cases, if bothsilence boundaries surrounding a silence gap are close (e.g., bothwithin 500 milliseconds of an interceding scene boundary in the silencegap), the silence boundaries are collapsed and merged to the sceneboundary. In this example, if the scene boundary is far enough fromeither silence boundary, no adjustment is made.

These are just a few examples, and other implementations additionally oralternatively use other techniques for adjusting or even removingcandidate boundaries. For example, in certain contexts, such as videonarration where a scene boundary without an associated transcript word(e.g., a scene boundary coinciding with a non-speech segment) mightindicate a useful location for a video segment boundary, someembodiments snap speech boundaries to proximate non-speech boundaries(e.g., scene boundaries, tool boundaries). In some cases, to avoidplacing boundaries or cuts in the middle of speech (e.g., words),candidate boundaries (e.g., scene boundaries, tool boundaries) that fallwithin a speech segment are removed. In some contexts such as video withbackground music, it may be desirable to allow boundaries or cuts in themiddle of speech (e.g., song vocals), so in some embodiments, some orall candidate boundaries that fall within a speech segment are notremoved. In some embodiments, tool boundaries derived from software toolselections, transitions, and/or uses that fall within a short silence orother non-speech segment (e.g., less than one second) are snapped to theclosest non-tool boundary. As such, in various embodiments, the type ofadjustment rule is tailored based on the context, and/or cut costs forcertain boundaries are defined or weighted appropriately, as describedin more detail below.

In order to compute a hierarchical video segmentation, some embodimentscompute and/or assign one or more cut costs for candidate boundaries.For example, as explained in more detail below, in order to compute anoptimal segmentation for one or more levels of the hierarchicalsegmentation, in some embodiments, a cost function is defined and/orevaluated for a candidate segmentation using one or more cut costsassociated with each candidate boundary associated with the candidatesegmentation. In some cases, cut costs for each candidate boundary arecomputed and assigned to each candidate boundary (e.g., as metadata)prior to computing the hierarchy. Examples of different types of cutcosts for candidate boundaries include VAD cut cost, silence cut cost,sentence cut cost, scene cut cost, tool cut cost, and/or others. In someembodiments, for some or all cut costs used in the cost function, a lowcut cost for a particular candidate boundary means the boundary would bea good location to cut the video or otherwise define a boundary for avideo segment.

For example, in some embodiments, a VAD cut cost is assigned to, orotherwise determined for, candidate boundaries. Depending on theembodiment, the VAD cut cost for a candidate boundary is a VAD score(e.g., a computed, normalized, and/or smoothed VAD score) of the audioof the video at the time of the boundary. In some embodiments wherehigher VAD scores indicate the presence of speech, using a VAD score asa VAD cut cost in the cost function discourages segmentation atboundaries located during speech.

In another example, in some embodiments, one or more silence cut costsare assigned to, or otherwise determined for candidate boundaries. Toencourage placing cuts at the beginning or end of long silence gaps(e.g., or other non-speech segments), the duration of a silence gap thatis adjacent to (e.g., preceding and/or following) a candidate boundaryis used to determine a silence cut cost for the boundary. In someembodiments, the silence duration is normalized, for example, by themaximum length of a video segment (e.g., maximum length of a clip atom,pre-defined target maximum length of a video segment in a particularhierarchy level). In some implementations, silence cut cost is inverselyproportional to the duration of an adjacent silence. Therefore, arelatively long silence duration results in a relatively low silence cutcost, encouraging segmentation at boundaries that are adjacent to longsilences. On the other hand, a relatively short silence duration resultsin a relatively large silence cut cost, discouraging segmentation atboundaries that are adjacent to short silences, thereby encouragingshort silence segments to be merged with adjacent non-silence segments.In some embodiments that pre-compute a portion of the cost functionprior to computing the hierarchy, each candidate boundary is assignedone or more values, such as silence duration(s) (e.g., for precedingsilence, subsequent silence, maximum adjacent silence, zero if there isno adjacent silence), normalized silence duration(s), silence cutcost(s) (e.g., corresponding to multiple adjacent silence durations, forone or more hierarchy levels), some other intermediate value, and/orother values.

In another example, in some embodiments, a sentence cut cost is assignedto, or otherwise determined for candidate boundaries. For example, insome embodiments, a candidate boundary is assigned a sentence cut costof zero if it is a sentence boundary, and other types of boundaries(e.g., utterance boundaries or scene boundaries that do not coincidewith sentence boundaries) are assigned a sentence cut cost of one (orsome other normalized value). Thus, in some embodiments, using asentence cut cost in the cost function encourages segmentation atsentence boundaries and discourages segmentation at other types ofboundaries.

In another example, in some embodiments, a scene cut cost is assignedto, or otherwise determined for candidate boundaries. For example, insome embodiments, for each scene boundary, histogram similarity of thepixels in the two video frames adjacent to the boundary (before andafter) is computed and assigned as the scene cut cost for the sceneboundary. Generally, histogram similarity is computed using any knowntechniques, for example, by computing a histogram distribution of pixelintensity values for the two frames on either side of boundary, andcalculating the distance between the two histogram distributions usingany suitable distance metric (e.g., correlation, Chi-squared,intersection, Hellinger/Bhattacharyya distance, Euclidean distance,Chebyshev distance, Manhattan distance). To encourage segmentation atscene boundaries between video frames with more significant visualchanges, in some embodiments, scene cut cost is inversely proportionalto histogram similarity. In this manner, the more different the twoadjacent video frames are, the larger the histogram similarity (distancebetween their histogram distributions), and the lower the scene cutcost. In some implementations, other types of boundaries (e.g.,utterance and sentence boundaries that do not coincide with sceneboundaries) are assigned a scene cut cost of one (or some othernormalized value). Thus, in various embodiments, using a scene cut costin the cost function encourages segmentation at scene boundaries wherethe adjacent video frames are more visually different.

In some embodiments, a custom cut cost is defined for certain boundarytypes. For example, in some embodiments, tool boundaries derived fromtool events such as software tool selections, transitions, and/or usesare assigned a tool cut cost. In various implementations, the tool cutcost is defined to place more emphasis on tool boundaries for toolevents that have a longer gap until the next tool event, for example, bydefining tool cut cost to be inversely proportional to the durationbetween tool boundaries. In some embodiments, the duration between toolboundaries is normalized (e.g., by the maximum duration between toolboundaries). Additionally or alternatively, the tool cut cost is definedto place more emphasis on tool boundaries that signal a larger semanticchange using encoded importance values for software tools. For example,in some embodiments where tool boundaries indicate a selection, change,and/or use of a software tool in creative software (e.g., while screencapturing or screencasting video of software usage), the importance ofdifferent types of software tools is quantified and encoded. In someembodiments, importance values for software tools are normalized (e.g.,on [0.1]), with larger importance values indicating a larger semanticchange. In an example embodiment, layer changes or changes in anavigational menu are encoded to indicate a larger semantic shift invideo content. In another example embodiment, opening up software isscored with a higher tool importance value than drawing a stroke. Thus,in various embodiments, for each tool boundary, the tool that wasselected or used at that time is looked up (e.g., from a software usagelog), mapped to a corresponding importance value (e.g., based on apre-defined mapping), and the importance value is used to compute toolcut cost. Thus, in some embodiments, using a tool cut cost in the costfunction encourages segmentation at tool boundaries for tool events thathave a longer gap until the next tool event and/or at tool boundariesthat signal a larger semantic change in video content.

Having defined and/or computed various cut costs for candidateboundaries, in some embodiments, the detected and/or adjusted candidateboundaries are used as boundaries for a segmentation of the video. Forexample, in some embodiments, the candidate boundaries are combined,de-duplicated, and/or used to segment or otherwise define start and endpoints for clip atoms, the most granular segmentation of the video.Using the clip atoms, some embodiments generate a hierarchicalsegmentation of the video by hierarchically clustering the clip atomsinto video segments at multiple levels of granularity. To accomplishthis, in some embodiments, a set of the candidate boundaries is selectedto form a complete and disjoint set of video segments at each of aplurality of levels of a hierarchical segmentation. Selecting a set ofthe candidate boundaries for a particular level of the hierarchicalsegmentation can also be thought of as clustering a corresponding set ofthe clip atoms into the particular level of the hierarchicalsegmentation. In some cases, higher levels segment the video intocoarser segments, and lower levels segments the video into finersegments. In some embodiments, video segment boundaries in coarserhierarchy levels (e.g., longer clips) are a strict subset of videosegment boundaries in finer hierarchy levels (e.g., shorter clips).

Generally, depending on the implementation, any suitable segmentationand/or clustering technique is applied to generate segmentations at anynumber of hierarchy levels. The following is a non-limiting example of apotential hierarchical segmentation.

The lowest level (level 0) of the hierarchy is formed by the clip atoms(e.g., defined based on the detected and/or adjusted candidateboundaries). In this example, level 0 is the most granular segmentationof the video.

The next level (level 1) of the hierarchy is formed by merging shortnon-speech clip atoms (e.g., non-speech atoms that have a duration belowsome threshold, such as one second) with adjacent (e.g., preceding,subsequent) clip atoms. Additionally or alternatively, speech boundaries(e.g., word and/or utterance boundaries) that fall inside a sentence areremoved. As such, in some embodiments, level 1 is formed with videosegments such as sentence clips, non-speech clips (e.g., silence clips)that are longer than some threshold, and/or clips cut or otherwisedefined by scene boundaries.

From level 2 and up, an optimal video segmentation is computed using aline breaking algorithm (e.g., Knuth and Plass' line breakingalgorithm), a dynamic programming, or some other segmentation algorithmthat evaluates a cost function for candidate segmentations to compute anoptimal segmentation for a particular level of the hierarchy. In someembodiments, the segmentation algorithm is repeatedly applied atsuccessive levels of the hierarchy (e.g., level 2 and up), clusteringthe video segments from a particular level into coarser and longersegments until reaching a desired top level, for example, when theentire timeline becomes a single chunk. In this example, the output is amulti-level hierarchical representation of the video.

In some embodiments, the cost function for the segmentation algorithm isdefined for a candidate segmentation based on cut costs for associatedboundaries. More specifically, for a particular hierarchy level, a setof boundaries for a candidate segmentation are selected from the set ofclip atom boundaries (and/or from the set of video segment boundariesthat define a preceding hierarchy level), and the selected boundariesform candidate video segments for the candidate segmentation. In someembodiments, a cut cost is defined for each candidate segment, and thecut cost for the candidate segmentation is defined as the sum of the cutcosts for its candidate segments. Example cut costs for a candidatesegment include a length cut cost based on the length of the candidatesegment, cut costs assigned to boundaries of the candidate segment,consistency cut cost that penalizes candidate segments that containscene boundaries, and/or others. Additionally or alternatively tosumming cut costs for the candidate segments in a candidatesegmentation, in some embodiments, a cut cost for a candidatesegmentation is computed by summing the cut costs assigned to eachboundary in the candidate segmentation. These are just a few examples,and other cost functions may be implemented within the scope of thepresent disclosure.

In some embodiments, to encourage minimum and maximum lengths for videosegments at a particular hierarchy level, a length cut cost is definedbased on pre-defined target minimum and maximum lengths. In one exampleimplementation, if the length of a candidate segment is within thetarget length range, length cut cost is zero. If the length of acandidate segment is larger than a pre-defined target maximum length fora given hierarchy level, the length cut cost for the candidate segmentis proportional to the length of the segment (e.g., and normalized bythe pre-defined target maximum length for the level). If the length of acandidate segment is less than a pre-defined target minimum length for agiven hierarchy level, the length cut cost for the candidate segment isset to some arbitrary value (e.g., a relatively large constant). Assuch, in this example, the length cut cost penalizes segmentations withcandidates segments that have durations outside a pre-defined targetrange.

In some embodiments, the boundary cost of a candidate segment is aweighed sum of boundary costs (e.g., silence, sentence and scene cutcosts) assigned to each boundary associated with the candidate segment.Depending on the embodiment and/or the type of boundary cut cost, theboundaries associated with a candidate segment used to compute theboundary cut cost are the end points of the candidate segment, the clipatom boundaries enclosed by the candidate segment, and/or both. In someimplementations, the boundary costs for a candidate segment (e.g.,silence, sentence, scene cut costs, tool cut costs) are weighted tofavor certain boundaries, such as boundaries (e.g., sentence boundaries,scene boundaries) that are adjacent to a long silence. In some cases,scene boundaries are favored since, in certain contexts, sceneboundaries usually signal larger semantic shift in the video content. Insome embodiments, the boundary cut cost for a candidate segmentation isthe sum of the boundary cut costs computed for each of its candidatesegments. Additionally or alternatively, the boundary cut cost for acandidate segmentation is the sum of the boundary cut costs assigned toeach boundary in the candidate segmentation.

In some embodiments, the consistency cost of a candidate segment isgiven by the sum of the scene cut costs for all clip atoms boundariesthat fall within the candidate segment (e.g., excluding the clip atomboundaries at the start and end points of the candidate segment). Thisconsistency cost effectively penalizes candidate segments that containscene boundaries in somewhere in within the candidate segment.

As such, in various implementations, the cost function of a segmentationalgorithm (e.g., a line breaking algorithm, a dynamic programmingalgorithm) computes multiple cut costs for a candidate segment, combinesthe multiple cut costs to compute a total cut cost for the candidatesegment, and/or sums the cut costs for the candidate segments in acandidate segmentation. In this manner, the line breaking algorithmevaluates candidate segmentations and identifies, for example, thecandidate segmentation that minimizes the cost function as the optimalvideo segmentation at a particular hierarchy level (e.g., level 2 andup). In an example implementation, the input into the segmentationalgorithm is the segmentation (e.g., a list of boundaries) from aprevious level of the hierarchy, and the segmentation algorithmidentifies an optimal segmentation for the next hierarchy level byevaluating the cost function for sets of boundaries sampled from theprevious level. In some embodiments, a segmentation at a given hierarchylevel is represented by a list of IDs and/or time values associated with(i) clip atom boundaries that define the segmentation, (ii) clusters ofclip atoms that form the video segments for the segmentation, and/or(iii) the video segments for the segmentation. In some implementations,the segmentation algorithm is iteratively applied to computesegmentations for successive levels of the hierarchy, for example, untilthe number of video segments in a hierarchy level is smaller than somenumber (e.g., 10), until the segmentation algorithm returns a singlechunk for a hierarchy level, and/or other criteria. As such, in variousembodiments, a hierarchical segmentation is computed with a plurality oflevels, where each successive level segments the video into videosegments with an increasing (or decreasing) amount of granularity.

In various embodiments, the hierarchical segmentation is representedusing one or more data structures. In an example implementation, thehierarchical segmentation is represented using a two dimension array,where the dimensions of the array correspond to the different levels ofthe hierarchy, and the values stored in each dimension of the arrayrepresent the video segments in a corresponding hierarchy level. In somecases, video segments are represented by values representing, orreferences to, timeline locations (e.g., startTime and/or endTime, forexample, in milliseconds), clip atoms (e.g., IDs), clip atom boundaries(e.g., IDs), and/or other representations. In some cases, a single copyof the video and a representation of boundary locations are maintained.In some embodiments, separate copies of video and/or separate copies ofthe video segments (e.g., chunks) are maintained for each level of thehierarchy. Generally, embodiments that maintain separate copies providefor faster access, scrubbing, trimming, and/or the like. These are justa few examples, and other representations may be implemented within thescope of the present disclosure.

In some embodiments, ingesting a video includes extracting metadataabout the video. Examples of different types of metadata extractioninclude transcribing associated audio, visually extracting content oraction tags from video (e.g., by performing object detection, forexample, using one or more neural networks), extracting software logevents from an associated temporal log (e.g., software usage log, suchone generated while screencasting an ADOBE BEHANCE® live stream), and/orothers. In some cases, transcribed audio is stored or otherwiseassociated with a corresponding video segment (e.g., in the hierarchicalsegmentation). Additionally or alternatively, transcribed audio isanalyzed for term frequency, and some or all terms (e.g., the mostfrequent n terms) are stored as searchable metadata tags associated withcorresponding video segments. As such, in some embodiments, an audiotranscript, keywords from an audio transcript, visually extractedcontent or action tags, action tags corresponding to extracted softwareevents, and/or other extracted features are stored, associated withcorresponding locations on the video timeline (or otherwise associatedwith corresponding video segments), and used as searchable metadata.Generally, extracting video features from video segments and using theextracted features as searchable metadata makes selecting and browsingvideo segments easier, as explained in more detail below.

Interacting with Hierarchical Clusters of Video

In some embodiments, a user interface provides one or more interactionelements that provide an interaction modality for selecting, navigating,playing, and/or editing a video based on a hierarchical segmentation ofa video. As explained above, a hierarchical segmentation of a videohierarchically clusters clip atoms (the smallest interaction unit of thevideo) into video segments at multiple levels of granularity. Ratherthan simply interacting with the video based on selections of particularvideo frames or time ranges, various implementations provide one or moreinteraction elements that allow users to interact with higher levelsemantic chunks of the video (the hierarchical clusters). Exampleinteractions include selecting, searching, playing, and/or editingparticular video segments (e.g., clusters of clip atoms) represented bythe hierarchical segmentation. Example interaction elements include avideo timeline segmented by the boundaries of the hierarchical clusters,a zoom bar for zooming in and out of the hierarchical clusters, a scrollbar for scrolling across the hierarchical clusters, a metadata panelshowing transcribed audio and extracted metadata tags for eachhierarchical cluster, and/or a search bar for searching extractedmetadata tags of the hierarchical clusters, to name a few possibilities.

For example, in some embodiments, a video timeline corresponding to thelength of the video is segmented by the boundaries of the hierarchicalclusters. In some cases, an interaction element allows a user to selecta level of the hierarchy, and the boundaries for the corresponding levelof the hierarchy are used to segment the video timeline into a set ofvideo segments defined by the level. When the user selects a portion ofthe video timeline, in some embodiments, the selection snaps to theboundaries of a corresponding video segment (e.g., cluster of clipatoms) defined by an active level of the hierarchy. For example, if theuser clicks or taps on a video segment, the video segment is selected(or de-selected). In another example, if the user clicks and drags (ortaps, holds, and drags) across multiple video segments on the videotimeline, the drag operation adds video segments to the selection (e.g.,as an expanding selection crosses a corresponding boundary) or removesvideo segments from a selection (e.g., as a decreasing selection crossesa corresponding boundary). Thus, a user can drag across the videotimeline to make a selection that snaps to video segment (cluster)boundaries. In some embodiments, when the user navigates to a differenthierarchy level, the video timeline and the selection of video segmentsare transformed or otherwise updated to reflect the boundaries of theselected hierarchy level, allowing for a refined selection of a portionof the video through selection of video segments with different levelsof granularity.

In some embodiments, a zoom bar and/or a scroll bar is provided tocontrol a window view of the video timeline. For example, in someembodiments, the zoom bar includes a thumb (or bar) that can be draggedalong a track (or trough). In some cases, the thumb has independentlymoveable (e.g., draggable) endpoints that control a correspondinglocation on the video timeline presented in the video timeline window.Thus, in some embodiments, resizing the thumb zooms in and out of thevideo timeline, and/or dragging the thumb along the track scrolls thevideo timeline through the timeline window.

In some embodiments, a metadata panel presents metadata (e.g.,transcribed audio and extracted metadata tags) for each video segment(cluster of clip atoms). In some cases, the metadata panel includes acomposite list of the metadata for all video segments, and the compositelist is segmented into metadata segments at locations that correspond tothe boundaries of the level of the hierarchy being viewed. In someembodiments, each of the metadata segments is independently selectable,which emphasizes (e.g., highlights) the selected metadata segment,emphasizes the corresponding video segment on the video timeline, movesa cursor to the first video frame of the corresponding video segment,and/or displays the video frame in a video player. Similarly, in someembodiments, selecting a particular video segment on the video timelinehighlights the video segment on the video timeline, emphasizes acorresponding metadata segment in the metadata panel, moves a cursor tothe first video frame of the corresponding video segment, and/ordisplays the video frame in a video player. In some embodiments, whenthe user navigates to a different hierarchy level, the metadata paneland the selection of metadata segments are updated to reflect theboundaries of the selected hierarchy level, allowing for a refinedselection of a portion of the video through selection of correspondingmetadata segments with different levels of granularity.

In some embodiments, a search bar is provided for searching metadatatags and other extracted metadata. In some cases, a user enters one ormore search criteria such as keywords, and extracted metadata associatedwith the video segments (e.g., clusters of clip atoms) are searched formatches with the search criteria. Examples of extracted metadata includea transcript of speech in an audio track, (frequent) transcript terms,visually extracted content or action tags, extracted action tagscorresponding to extracted software events, and/or other extractedfeatures. In some embodiments, corresponding matching video segments(i.e., segments with matching metadata) are emphasized (e.g.,highlighted) on the video timeline, and/or corresponding matchingmetadata segments are emphasized (e.g., highlighted) in the metadatapanel. In some embodiments, when the user navigates to a differenthierarchy level, the video timeline and/or the metadata panel aretransformed or otherwise updated to reflect the boundaries of theselected hierarchy level, and the search results (matching videosegments and/or metadata segments) are updated based on the boundariesof the selected hierarchy level. Thus, in some embodiments, changing thelevel of hierarchy during an active search (e.g., with highlightedsearch results) can provide search results with different levels ofgranularity, allowing for a more flexible and efficient searchexperience.

In some embodiments, different types of emphasis are applied todifferent selection states for video segments (e.g., clusters of clipatoms). For example, some embodiments may apply different types ofemphasis to unselected video segments, a video segment corresponding toa current playback position, a video or metadata segment being hoveredover, clicked or highlighted video or metadata segments, video ormetadata segments with metadata tags that match a keyword search, videosegments (and corresponding metadata segments) that have been added toan operational queue (e.g., a playback queue), some combination thereof,and/or others. Examples of different types of emphasis include differentcolors, gradients, patterns, outlines, shadows, and/or others.

Depending on the implementation, any number and variety of operationsare performed on selected video segments (and/or a corresponding portionof the video). For example, in various embodiments, based on a selectionof a corresponding interaction element(s), the selected video segmentsare played back (e.g., by playing only the selected video segments),trimmed (e.g., by removing the unselected video segments), edited insome other way (e.g., by rearranging, cropping, applying transitions oreffects, adjusting color, adding titles or graphics), exported, and/orother operations.

Example Video Editing Environment

Referring now to FIG. 1 , a block diagram of example environment 100suitable for use in implementing embodiments of the invention is shown.Generally, environment 100 is suitable for video editing or playback,and, among other things, facilitates hierarchical segmentation of videoand interactions with resulting hierarchical clusters of video segments.Environment 100 includes client device 110 and server 150. In variousembodiments, client device 110 and/or server 150 are any kind ofcomputing device capable of facilitating video editing or playback, suchas computing device 2600 described below with reference to FIG. 26 .Examples of computing devices include a personal computer (PC), a laptopcomputer, a mobile or mobile device, a smartphone, a tablet computer, asmart watch, a wearable computer, a personal digital assistant (PDA), amusic player or an MP3 player, a global positioning system (GPS) ordevice, a video player, a handheld communications device, a gamingdevice or system, an entertainment system, a vehicle computer system, anembedded system controller, a camera, a remote control, a bar codescanner, a computerized measuring device, an appliance, a consumerelectronic device, a workstation, some combination thereof, or any othersuitable computer device.

Environment 100 also includes storage 190. Storage 190 generally storesinformation including data, data structures, computer instructions(e.g., software program instructions, routines, or services), and/ormodels (e.g., machine learning models) used in some embodiments of thetechnologies described herein. In an embodiment, storage 190 comprises adata store (or computer data memory). Further, although depicted as asingle data store component, in some embodiments, storage 190 isimplemented as one or more data stores (e.g., a distributed storagenetwork) and/or in the cloud.

The components of environment 100 communicate with each other via anetwork 105. In some embodiments, network 105 includes one or more localarea networks (LANs), wide area networks (WANs), and/or other networks.Such networking environments are commonplace in offices, enterprise-widecomputer networks, intranets, and the Internet.

In the example illustrated in FIG. 1 , client device 110 includes videointeraction engine 120, and server 150 includes video ingestion tool155. In various embodiments, video interaction engine 120, videoingestion tool 155, and/or any of the elements illustrated in FIG. 1 areincorporated, or integrated, into an application(s) (e.g., acorresponding application on client device 110 and server 150,respectively), or an add-on(s) or plug-in(s) to an application(s). Insome embodiments, the application(s) is any application capable offacilitating video editing or playback, such as a stand-aloneapplication, a mobile application, a web application, and/or the like.In some implementations, the application(s) comprises a web application,for example, that is accessible through a web browser, hosted at leastpartially server-side, and/or the like. Additionally or alternatively,the application(s) include a dedicated application. In some cases, theapplication is integrated into an operating system (e.g., as a service).Example video editing applications include ADOBE PREMIERE PRO and ADOBEPREMIERE ELEMENTS.

In various embodiments, the functionality described herein is allocatedacross any number of devices. In some embodiments, video editingapplication 115 is hosted at least partially server-side, such thatvideo interaction engine 120 and video ingestion tool 155 coordinate(e.g., via network 105) to perform the functionality described herein.In another example, video interaction engine 120 and video ingestiontool 155 (or some portion thereof) are integrated into a commonapplication executable on a single device. Although some embodiments aredescribed with respect to an application(s), in some embodiments, any ofthe functionality described herein is additionally or alternativelyintegrated into an operating system (e.g., as a service), a server(e.g., a remote server), a distributed computing environment (e.g., as acloud service), and/or otherwise. These are just examples, and anysuitable allocation of functionality among these or other devices may beimplemented within the scope of the present disclosure.

To begin with a high-level overview of an example workflow through theconfiguration illustrated in FIG. 1 , client device 110 is a desktop,laptop, or mobile device such as a tablet or smart phone, and videoediting application 115 provides a video editing interface. In somecases, a user records a video using video recording capabilities ofclient device 110 (or some other device) and/or some applicationexecuting at least partially on the device (e.g., ADOBE BEHANCE). Insome cases, a user accesses a video through video editing application115, and/or otherwise uses video editing application 115 to identify thelocation where a video is stored (whether local to client device 110, atsome remote location such as storage 190, or otherwise). In some cases,video editing application 115 uploads the video or otherwisecommunicates the location of the video to server 150, and videoingestion tool 155 performs one or more ingestion functions on thevideo. In some embodiments, video ingestion tool 155 (e.g., videosegmentation component 160 and/or hierarchical clustering component 170)creates a hierarchical segmentation of the video, for example, byidentifying segment boundaries for hierarchical clusters of videosegments, generating a representation of a hierarchical segmentation ofthe video, and/or breaking up the video into corresponding videosegments. In another example, video ingestion tool 155 (e.g., metadataextraction component 175) extracts metadata about the video, forexample, by transcribing associated audio, visually extracting contentor action tags from video (e.g., using one or more neural networks),extracting software log events from an associated temporal log, and/orother ways). In some cases, video editing application 115 and/or videoingestion tool 155 store the video (e.g., as one of video files 192),clip atoms of the video (e.g., clip atoms 194), video segments formed byhierarchical clusters of the clip atoms (e.g., hierarchical clusters196), segment boundaries for clip atoms and/or higher level videosegments (e.g., segment boundaries 198), and/or some representationthereof in any suitable storage location, such as storage 190, clientdevice 110, server 150, some combination thereof, and/or otherlocations.

In some embodiments, once a video is ingested, video editing application115 (e.g., video interaction engine 120) provides a user interface withone or more interaction elements that allow a user to interact with theingested video, and more specifically, with hierarchical clusters ofvideo segments of the video. Some non-limiting examples of interactionelements include a video timeline segmented by the boundaries of thehierarchical clusters (e.g., segmented timeline tool 125), a zoom barfor zooming in and out of the hierarchical clusters (e.g., zoom/scrollbar tool 130), a scroll bar for scrolling across the hierarchicalclusters (e.g., zoom/scroll bar tool 130), a metadata panel showingtranscribed audio and extracted metadata tags for each hierarchicalcluster (e.g., metadata panel tool 135), a search bar for searchingextracted metadata tags of the hierarchical clusters (e.g., search tool140), one or more editing tools for operating on selected video segments(e.g., video edit tool 145), and/or a playback window that plays backselected video segments (e.g., video playback tool 148), to name a fewpossibilities. Thus, in various embodiments, video interaction engine120 provides a user interface that allows a user to select, navigate,play, and/or edit a video based on interactions with hierarchicalclusters of video segments.

In the sections that follow, the example workflow through theconfiguration illustrated in FIG. 1 is described in more detail,starting with video ingestion and followed by various ways ofinteracting with the video.

Ingestion, Segmentation, and Hierarchical Clustering of Video

Continuing with the preceding example, in some embodiments, videoingestion tool 155 ingests a video (e.g., a video file, a portion of avideo file, video represented or otherwise identified by a projectfile). In some embodiments, ingesting a video includes generating ahierarchical segmentation of the video that identifies clip atoms of thevideo (e.g., clip atoms 194), video segments formed by hierarchicalclusters of the clip atoms (e.g., hierarchical clusters 196), and/orsegment boundaries for clip atoms and/or higher-level video segments(e.g., segment boundaries 198). Additionally or alternatively, in someembodiments, ingesting a video includes extracting metadata about thevideo and associating the extracted metadata with corresponding portionsof the video (e.g., corresponding clip atoms, higher-level videosegments, portions of the video timeline).

In the example illustrated in FIG. 1 , video ingestion tool 155 includesvideo segmentation component 160, hierarchical clustering component 170,and metadata extraction component 175. In an implementation, videosegmentation component 160 computes an over-segmentation of the video'stimeline to identify boundaries for clip atoms of the video, andhierarchical clustering component 170 hierarchically clusters the clipatoms to form a hierarchical segmentation of the video. Metadataextraction component 175 extracts and associates metadata about thevideo with corresponding video segments (e.g., clip atoms and/orhigher-level video segments).

Generally, video segmentation component 160 computes over-segmentationof the video timeline of an identified video (e.g., being ingested). Inthe example illustrated in FIG. 1 , video segmentation component 160includes candidate boundary detection component 162, boundary adjustmentcomponent 164, and cut cost computation component 166. In this example,candidate boundary detection component 162 applies one or more detectiontechniques to detect candidate boundaries for clip atoms, boundaryadjustment component 164 applies one or more adjustment techniques toadjust the candidate boundaries to finalize the clip atom boundaries,and video segmentation component 160 generates or otherwise stores arepresentation of the clip atom boundaries and/or the corresponding clipatoms. In order to support hierarchical clustering (e.g., byhierarchical clustering component 170), in some embodiments, cut costcomputation component 166 computes one or more cut costs for candidateboundaries and associates candidate boundaries with associated cutcosts. In some embodiments, the resulting representation of the clipatom boundaries, corresponding clip atoms, and/or corresponding cutcosts forms the over-segmentation of the video timeline.

In some embodiments, candidate boundary detection component 162 detectscandidate boundaries for clip atoms (and higher-level video segments)using one or more detection techniques. In various embodiments,candidate boundary detection component 162 uses any number andcombination of detection techniques to identify speech boundaries (e.g.,utterance boundaries, word boundaries, sentence boundaries), sceneboundaries, event boundaries for software log events, and/or other typesof boundaries.

For example, in some embodiments, candidate boundary detection component162 identifies speech boundaries from a transcript of an audio trackassociated with the video. In some cases, candidate boundary detectioncomponent 162 applies any known speech-to-text algorithm to generate atranscript, detect speech segments (e.g., corresponding to words,sentences, utterances of continuous speech separated by audio gaps,etc.), detect non-speech segments (e.g., pauses, silence, or non-speechaudio), and/or the like. In some cases, candidate boundary detectioncomponent 162 parses the transcript into speech segments and/ornon-speech segments, for example, by applying natural languageprocessing based on linguistic features of the transcript (e.g., usingboundary detection logic), using a natural language processing model(e.g., a machine learning model), some other kind of segmentationtechnique, and/or other techniques. In some embodiments, candidateboundary detection component 162 associates the transcript with thetimeline of the video, maps detected speech (and/or non-speech) segmentsto locations on the video timeline, and identifies locations ofcandidate speech boundaries (e.g., utterance boundaries, wordboundaries, sentence boundaries) at the start and end of correspondingspeech segments. In some embodiments, to segment the video's timelineinto a complete and disjoint set of speech segments, gaps between speechsegments are considered to be speech segments (e.g., with a silencelabel applied) and/or considered to be silence segments (or silencegaps).

In some embodiments, candidate boundary detection component 162 detectsscene boundaries from video frames of the video. A scene boundary (alsocalled a shot boundary) is a video cut or other detectible scenetransition in the video. In some cases, a video cut is a hard cutbetween two adjacent video frames, a multi-frame video cut that spans asequence of multiple video frames (e.g., a fade or a wipe), and/or othertypes. Depending on how a video was generated, a scene boundary can becreated by recording consecutive takes, by concatenating two differenttakes (e.g., using video editing software), by generating a transition(e.g., using video editing software), and/or other ways.

In some embodiments, candidate boundary detection component 162 detectsscene boundaries by detecting abrupt changes in video frames, forexample, using any known technique. In some situations where a detectedscene boundary is based on a video cut between two adjacent videoframes, candidate boundary detection component 162 identifies the sceneboundary at a location on the video timeline between the video frames.In some embodiments where a detected scene boundary is based on amulti-frame video cut that spans a sequence of more than two videoframes, candidate boundary detection component 162 identifies the sceneboundary at a location on the video timeline that is centered (or atsome other location) in the sequence of video frames. These and otherpossible ways to detect scene boundaries (e.g., video cuts) within thescope of present disclosure are described in co-pending U.S. applicationSer. No. 16/879,362, filed on May 20, 2020, the contents of which arehereby incorporated by reference in their entirety.

In some embodiments, candidate boundary detection component 162identifies the event boundaries from log events represented in one ormore temporal logs, such as software usage logs associated with thevideo. For example, in some embodiments, candidate boundary detectioncomponent 162 accesses one or more temporal logs associated with thevideo, and detects log events and corresponding event boundaries basedon the logs.

Various implementations involve different types of temporal logs and/orlog events. For example, in one implementation involving screen capturedor screencast videos of tutorials for creative software such as ADOBEPHOTOSHOP or ADOBE FRESCO, a software usage log generated by thecreative software while screen capturing or screencasting is read toidentify the times of event boundaries when detected log events such astool events (e.g., indicating a selection, change, or use of aparticular software tool, such as select brush, create layer, etc.)occurred. In an example gaming implementation, a software usage log isread to identify the times of event boundaries for detected software logevents such as leveling up or beating an enemy. In an example cookingimplementation, a usage log is read to identify the times of eventboundaries for logged events (e.g., manually, visually detected) in thevideo such as a user picking up a different pot or cooking tool.Although the foregoing examples involve temporal logs with log eventsderived from video frames, this need not be the case. For example, in animplementation with a live chat or chat stream associated with alivestreamed video, a corresponding user chat log or session is read toidentify times of event boundaries such as chat messages about aparticular topic. In an example video streaming implementation (whetherlivestreaming or viewing archived video), a usage log representing how auser(s) has viewed the video is read to identify the times of eventboundaries for detected interaction events such as navigational events(e.g., play, pause, skip). Generally, any type of temporal log and/ormetadata can be read to identify times for event boundaries.

In some cases, if a log does not report times relative to the videotimeline, candidate boundary detection component 162 converts, maps, orotherwise associates an event timeline represented in the log with thevideo timeline. As such, in some embodiments, candidate boundarydetection component 162 extracts the times of log events captured by alog (e.g., by reading from structured data fields, by applying searchpatterns, natural language processing, and/or other rules tounstructured data, etc.), maps the times of the log events to locationson the video timeline (if necessary), and identifies event boundaries(e.g., tool boundaries identifying locations of tool events) at thecorresponding locations on the video timeline.

In some embodiments, boundary adjustment component 164 adjusts thelocations of detected candidate boundaries using one or more adjustmenttechniques. In some cases, certain types of candidate boundaries areadjusted using certain types of adjustment techniques. In variousimplementations, boundary adjustment component 164 adjusts candidateboundaries derived from an audio track of the video that includes speech(i.e., speech boundaries, such as utterance or sentence boundaries)using voice activity detection (VAD), adjusts speech boundaries bysnapping them to proximate scene boundaries (e.g., when a scene boundaryfalls within a silence), adjusts event boundaries derived from logevents (e.g., by snapping tool boundaries that fall within a shortsilence, such as less than one second, to the closest non-toolboundary), and/or using other adjustment techniques.

In some embodiments, to avoid placing a video segment boundary (e.g., aclip atom boundary) in the middle of speech, boundary adjustmentcomponent 164 adjusts candidate boundaries for video segments using VAD.VAD is an audio technique that detects the presence (or likelihood ofthe presence) of human voice in an audio signal at a particular time. Insome embodiments, VAD outputs VAD scores that are normalized to acontinuous range, for example, on [0, 1] such that VAD=1 means there ishuman voice and VAD=0 means no human voice. As such, in some cases,boundary adjustment component 164 computes VAD scores for an associatedaudio track of the video, and uses the VAD scores at locations ofcandidate boundaries as a cut cost to identify adjustments that placethe candidate boundaries in optimal locations (e.g., by adjustingboundaries with high VAD cost and/or permitting boundaries with low VADcost). For example, in some cases, boundary adjustment component 164refines candidate speech boundaries by snapping the candidate boundariesto proximate locations within a neighborhood of each boundary where VADscores are at a minimum. In another example, boundary adjustmentcomponent 164 closes a silence gap between two speech chunks (e.g., thatis shorter than some duration d, such as one second) by searching thesilence gap for the lowest VAD score and merging the candidate speechboundaries surrounding the silence gap into an adjusted boundary at thelocation of the lowest VAD score.

FIGS. 2A and 2B are illustrations of example boundary adjustments forvideo segments based on voice activity detection. For example, FIG. 2Ashows speech segment 210 defined by boundaries b₁ and b₂, and a plot ofVAD scores 215 indicating the presence of human voice with respect tospeech segment 210. For each boundary b₁ and b₂, boundary adjustmentcomponent 164 searches a neighborhood r (e.g., on both sides) of theboundary for the location at which VAD score 215 is minimized (e.g.,indicating the absence of human voice). The resulting locations are usedas the locations for updated boundaries b₁* and b₂* for adjusted speechsegment 220.

FIG. 2B shows speech segment 230 with right boundary b₁ and speechsegment 240 with left boundary b₁, separated by a silence gap withduration d, a plot of VAD scores 250 indicating the presence of humanvoice with respect to speech segments 230 and 240. In this example,boundary adjustment component 164 determines whether the silence gap isless than (or equal to) some threshold duration, and if so, searches thesilence gap for the location at which VAD score 215 is minimized.Boundary adjustment component 164 merges the two boundaries b₁ and b₂,in FIG. 2B into a new boundary b* between adjusted speech segments 235and 245.

In some embodiments, boundary adjustment component 164 applies smoothingto the VAD scores prior to adjusting candidate speech boundaries.Instead of cutting or otherwise defining a video segment boundary rightat the end of the speech, smoothing the VAD scores prior to snappingboundaries to local VAD minima effectively adds a temporal buffer tospeech boundaries, thereby cutting or otherwise defining boundaries forspeech chunks at some distance away from the unsmoothed boundaries,which generates more natural transitions between video segments. In anexample embodiment, boundary adjustment component 164 applies any knownVAD technique to some or all of an audio track associated with video tocalculate VAD scores, applies smoothing to the VAD scores (e.g., using akernel such as a Gaussian or Cauchy kernel, by applying an audio filter,etc.), and adjusts the locations of candidate speech boundaries based onthe smoothed VAD scores. In some embodiments, smoothing is only appliedto the VAD scores at the location of (e.g., centered around) candidatespeech boundaries. In some cases, the width of a smoothing kernel orfilter corresponds with, or sets a minimum value for, the neighborhoodto search. Since smoothing a signal generally leaves a tail, and thesize of the tail generally corresponds to the size of the kernel orfilter (e.g., width r), some embodiments set the neighborhood to searchgreater than or equal to the size of the kernel, filter, or tail (e.g.,for each audio-derived candidate boundary such as each candidate speechboundary, search the lowest VAD value in a neighborhood of width r).These are just a few examples, and other embodiments apply additional oralternative smoothing techniques.

In some embodiments, boundary adjustment component 164 adjusts candidatespeech boundaries by snapping the candidate speech boundaries toproximate scene boundaries. Since in some embodiments, speech boundaries(e.g., utterance boundaries, sentence boundaries) are derived from audioassociated with the video (e.g., transcribing speech from the audio anddetecting utterance, word, and/or sentence boundaries from thetranscript), while scene boundaries are derived from video frames of thevideo, there can be slight discrepancies between the two types ofboundaries. If the video is cut (or a boundary placed) just before orafter a scene boundary, it causes a jarring effect (a sudden jump at thebeginning or end of the cut or segment). To avoid such jarring cuts andto account for discrepancies, in some embodiments, the locations of someor all audio-derived candidate boundaries (e.g., speech boundaries) areadjusted.

In an example implementation, boundary adjustment component 164 detectsscene boundaries that fall within a silence or other non-speech segment(e.g., based on looking up and finding no word at a correspondingportion of an associated transcript, based on an associated VAD scorebeing below some threshold value), and boundary adjustment component 164snaps proximate speech and/or silence boundaries to the detected sceneboundaries. In this example, since detected scene boundaries are locatedwithin a silence gap, a proximate speech boundary is can also be thoughtof as a silence boundary for the silence gap. Thus, in some embodiments,silence boundaries that are proximate to (e.g., within some neighborhoodsuch as 500 milliseconds of) a scene boundary that falls within asilence gap are snapped to the scene boundary. In some cases, if bothsilence boundaries surrounding a silence gap are close (e.g., bothwithin 500 milliseconds of an interceding scene boundary in the silencegap), the silence boundaries are collapsed and merged to the sceneboundary. On the other hand, in some embodiments, if the scene boundaryis far enough from either silence boundary, no adjustment is made.

These are just a few examples, and in other implementations, boundaryadjustment component 164 additionally or alternatively uses othertechniques to adjust or even remove candidate boundaries. For example,in certain contexts, such as video narration where a scene boundarywithout an associated transcript word (e.g., a scene boundary coincidingwith a non-speech segment) might indicate a useful location for a videosegment boundary, some embodiments snap speech boundaries to proximatenon-speech boundaries (e.g., scene boundaries, tool boundaries). In somecases, to avoid placing boundaries or cuts in the middle of speech(e.g., words), candidate boundaries (e.g., scene boundaries, toolboundaries) that fall within a speech segment are removed. In somecontexts such as video with background music, it may be desirable toallow boundaries or cuts in the middle of speech (e.g., song vocals), soin some embodiments, some or all candidate boundaries that fall within aspeech segment are not removed. In some embodiments, tool boundariesderived from software tool selections, transitions, and/or uses thatfall within a short silence or other non-speech segment (e.g., less thanone second) are snapped to the closest non-tool boundary. As such, invarious embodiments, the type of adjustment rule implemented by boundaryadjustment component 164 is tailored based on the context, and/or cutcosts for certain boundaries are defined or weighted appropriately, asdescribed in more detail below.

Accordingly, candidate boundary detection component 162 and/or boundaryadjustment component 164 identifies candidate boundaries (e.g.,locations on the video timeline) for an over-segmentation of the video.In order to support hierarchical clustering (e.g., by hierarchicalclustering component 170), in some embodiments, cut cost computationcomponent 166 computes one or more cut costs for candidate boundariesand associates candidate boundaries with corresponding cut costs. Forexample, as explained in more detail below, in order to compute anoptimal segmentation for one or more levels of a hierarchicalsegmentation, in some embodiments, a cost function is defined and/orevaluated using one or more cut costs associated with each candidateboundary. In some cases, prior to computing the hierarchy, cut costcomputation component 166 computes the cut costs and associatescorresponding values with each candidate boundary (e.g., as metadata).Examples of different types of cut costs for candidate boundariesinclude VAD cut cost, silence cut cost, sentence cut cost, scene cutcost, tool cut cost, and/or others. Depending on the implementation, forsome or all cut costs, a low cut cost for a particular candidateboundary means the boundary would be a good location to cut the video orotherwise define a boundary for a video segment.

For example, in some embodiments, cut cost computation component 166determines and/or assigns a VAD cut cost for candidate boundaries. Insome cases, the VAD cut cost for a candidate boundary is a VAD score(e.g., a computed, normalized, and/or smoothed VAD score) at the timewhere the boundary is located. In some embodiments where higher VADscores indicate the presence of speech, using a VAD score as a VAD cutcost in the cost function discourages segmentation at boundaries locatedduring speech.

In another example, in some embodiments, cut cost computation component166 determines and/or assigns one or more silence cut costs forcandidate boundaries. To encourage placing cuts at the beginning or endof long silence gaps (e.g., or other non-speech segments), cut costcomputation component 166 uses the duration of a silence gap that isadjacent to (e.g., preceding and/or following) a candidate boundary todetermine a silence cut cost for the boundary. In some embodiments, thesilence duration is normalized, for example, by the maximum length of avideo segment (e.g., maximum length of a clip atom, pre-defined targetmaximum length of a video segment in a particular hierarchy level). Forexample, in some embodiments, a silence cut cost is defined as1.0−dur_(silence)/MAXLENGTH, such that a relatively long silenceduration results in a relatively low silence cut cost, encouragingsegmentation at boundaries that are adjacent to long silences. On theother hand, a relatively short silence duration results in a relativelylarge silence cut cost, discouraging segmentation at boundaries that areadjacent to short silences, thereby encouraging short silence segmentsto be merged with adjacent non-silence segments. In some embodimentsthat pre-compute a portion of the cost function prior to computing thehierarchy, cut cost computation component 166 assigns to each candidateboundary one or more values, such as silence duration(s) (e.g., forpreceding silence, subsequent silence, maximum adjacent silence, zero ifthere is no adjacent silence), normalized silence duration(s), silencecut cost(s) (e.g., corresponding to multiple adjacent silence durations,for one or more hierarchy levels), some other intermediate value, and/orother values.

In another example, in some embodiments, cut cost computation component166 determines and/or assigns a sentence cut cost for candidateboundaries. For example, in some embodiments, a candidate boundary isassigned a sentence cut cost of zero if it is a sentence boundary, andother types of boundaries (e.g., utterance boundaries or sceneboundaries that do not coincide with sentence boundaries) are assigned asentence cut cost of one (or some other normalized value). Thus, in someembodiments, using a sentence cut cost in the cost function encouragessegmentation at sentence boundaries and discourages segmentation atother types of boundaries.

In another example, in some embodiments, cut cost computation component166 determines and/or assigns a scene cut cost for candidate boundaries.For example, in some embodiments, for each scene boundary, cut costcomputation component 166 computes histogram similarity of the pixels inthe two video frames adjacent to the boundary (before and after) andassigns the resulting value as the scene cut cost for the sceneboundary. Generally, histogram similarity is computed using any knowntechniques, for example, by computing a histogram distribution of pixelintensity values for the two frames on either side of boundary, andcalculating the distance between the two histogram distributions usingany suitable distance metric (e.g., correlation, Chi-squared,intersection, Hellinger/Bhattacharyya distance, Euclidean distance,Chebyshev distance, Manhattan distance). To encourage segmentation atscene boundaries between video frames with more significant visualchanges, some embodiments use the inverse of histogram similarity tocompute scene cut cost. For example, in some embodiments, scene cut costcost_(scene) is given by:cost_(scene)=1.0−histogram_similarity/(frame_height*frame_width)  (1)In this example, the more different the two adjacent video frames are,the larger the histogram_similarity (distance between their histogramdistributions), and the lower the scene cut cost. In someimplementations, other types of boundaries (e.g., utterance and sentenceboundaries that do not coincide with scene boundaries) are assigned ascene cut cost of one (or some other normalized value). Thus, in variousembodiments, using a scene cut cost in the cost function encouragessegmentation at scene boundaries where the adjacent video frames aremore visually different.

In some embodiments, cut cost computation component 166 determinesand/or assigns a custom cut cost for certain boundary types. Forexample, in some embodiments, cut cost computation component 166determines and/or assigns a tool cut cost for tool boundaries derivedfrom tool events identified from a software usage log (e.g., softwaretool selections, transitions, and/or uses). In some cases, tool cut costis defined to place more emphasis on tool boundaries for tool eventsthat have a longer gap until the next tool event, for example, by usingthe duration of time between tool boundaries to compute tool cut cost.In some embodiments, the duration between tool boundaries is normalized(e.g., by the maximum duration between tool boundaries for the video).

Additionally or alternatively, tool cut cost is defined to place moreemphasis on tool boundaries that signal a larger semantic change. Forexample, in some embodiments where tool boundaries corresponding toselections, changes, and/or uses of software tools, the importance ofeach software tool is quantified and encoded (e.g., onto a range) togenerate a (pre-determined) mapping of software tools to correspondingimportance values. In some embodiments, importance values for softwaretools are normalized (e.g., on [0.1]), with larger importance valuesindicating a larger semantic change. In an example embodiment, layerchanges or changes in a navigational menu are encoded to indicate alarger semantic shift in video content (e.g., relatively largerimportance values). In another example embodiment, opening up softwareis scored with a higher importance value than drawing a stroke. Thus, invarious embodiments, for each tool boundary, cut cost computationcomponent 166 identifies a corresponding software tool (e.g., from asoftware usage log), map the identified software tool to a correspondingimportance value (e.g., based on a pre-defined mapping), and uses theimportance value to compute tool cut cost.

In a non-limiting example embodiment that places more emphasis on toolboundaries for tool events that have a longer gap until the next toolevent and on tool importance, cut cost computation component 166computes tool cut cost as:cost_(tool)=(1.0−TimeToNextTool/max(TimeToNextTool))*tool_(Importance)  (2)where TimeToNextTool measures the time difference between a particulartool boundary and the next tool event, Max(TimeToNextTool) is themaximum value of TimeToNextTool across all tool boundaries, andtool_(importance) is a value between [0,1]. Thus, in some embodiments,using a tool cut cost in the cost function encourages segmentation attool boundaries for tool events that have a longer gap until the nexttool event and/or at tool boundaries that signal a larger semanticchange in video content.

As such, video segmentation component 160 identifies boundary locationsfor an over-segmentation of the video timeline and computes cut costsfor the boundaries. Hierarchical clustering component 170 uses theidentified boundaries and cut costs to compute a hierarchicalsegmentation of the video. Generally, depending on the implementation,any suitable segmentation and/or clustering technique is applied toidentify segment boundaries at any number of hierarchy levels. In oneexample, hierarchical clustering component 170 uses the detected and/oradjusted candidate boundaries identified by video segmentation component160 as boundaries for clip atoms for the video. For example, in someembodiments, the candidate boundaries are combined, de-duplicated,and/or used to segment or otherwise define start and end points for clipatoms, the most granular segmentation of the video. In some embodiments,hierarchical clustering component 170 uses the clip atoms (and/or clipatom boundaries) to generate a hierarchical segmentation of the video byhierarchically clustering and merging the clip atoms into video segmentsat multiple levels of granularity.

To accomplish this, in some embodiments, hierarchical clusteringcomponent 170 selects a set of the clip atom boundaries to form acomplete and disjoint set of video segments at each of a plurality oflevels of a hierarchical segmentation. Selecting a set of a set of theclip atom boundaries for a particular level of the hierarchicalsegmentation can also be thought of as clustering a corresponding set ofthe clip atoms into video segments for the particular level. FIG. 3illustrates an example hierarchical segmentation of a video, inaccordance with embodiments of the present invention. In FIG. 3 , higherlevels of the hierarchy segment the video into coarser segments, andlower levels segments the video into finer segments. In this example,level 0 segments the video into the finest granularity, the video's clipatoms. Furthermore, in FIG. 3 , video segment boundaries in coarserhierarchy levels (e.g., longer clips) are a strict subset of videosegment boundaries in finer hierarchy levels (e.g., shorter clips).

In some embodiments, hierarchical clustering component 170 uses the clipatoms as the finest (lowest) level of the hierarchical segmentation, andforms higher levels by merging clusters of consecutive clip atoms intolarger video segments (corresponding to larger chunks on the timeline).For example, in some cases, hierarchical clustering component 170 formsa level of the hierarchical segmentation (e.g., level 1) by mergingshort non-speech clip atoms (e.g., non-speech atoms that have a durationbelow some threshold, such as one second) with adjacent (e.g.,preceding, subsequent) clip atoms. Additionally or alternatively,hierarchical clustering component 170 forms a level of the hierarchicalsegmentation (e.g., level 1) by removing speech boundaries (e.g., wordand/or utterance boundaries) that fall inside a sentence. As such, insome embodiments, level 1 is formed with video segments such as sentenceclips, non-speech clips (e.g., silence clips) that are longer than somethreshold, and or clips cut or otherwise defined by scene boundaries.

In embodiments, a cost function is defined based on cut costs forsegment boundaries, and hierarchical clustering component 170 executes asegmentation algorithm to compute an optimal segmentation for one ormore levels of the hierarchy by evaluating the cost function forcandidate segmentations at each level. In some cases, hierarchicalclustering component 170 repeatedly applies a segmentation algorithmsuch as a line breaking or dynamic programming algorithm to successivelevels of the hierarchy (e.g., level 2 and up), clustering the videosegments from a particular level into coarser and longer segments untilreaching a desired top level, for example, when the entire timelinebecomes a single chunk. In this example, the output is a multi-levelhierarchical representation of the video.

In some embodiments, the cost function for the segmentation algorithm isdefined for a candidate segmentation based on cut costs for associatedboundaries. More specifically, for a particular hierarchy level,hierarchical clustering component 170 selects boundaries for a candidatesegmentation from the set of clip atom boundaries and/or from the set ofvideo segment boundaries that define a preceding hierarchy level), andthe selected boundaries form candidate video segments for the candidatesegmentation. In some embodiments, a cut cost is defined for eachcandidate segment, and the cut cost for the candidate segmentation isdefined as the sum of the cut costs for its candidate segments.Additionally or alternatively, a cut cost is defined for a candidatesegmentation as the sum of the cut costs for each boundary in thecandidate segmentation. Example cut costs for a candidate segmentationinclude length cut cost based on the length of candidate segments, cutcosts for boundaries in the candidate segmentation, consistency cut costthat penalizes candidate segments in a candidate segmentation thatcontain scene boundaries in within a candidate segment, and/or others.

In some embodiments, to encourage minimum and maximum lengths for videosegments at a particular hierarchy level, hierarchical clusteringcomponent 170 computes length cut cost based on pre-defined targetminimum and maximum lengths. In an example embodiment, hierarchicalclustering component 170 computes length cut cost for a candidatesegment as:

$\begin{matrix}{{cost}_{length} = \left\{ \begin{matrix}{\frac{{{length}({seg})} - {{MAX}{LENGTH}}}{{MAX}{LENGTH}},} & {{{if}{{length}({seg})}} > {{MAX}{LENGTH}}} \\{C_{MAXCOST},} & {{{if}{{length}({seg})}} < {{MIN}{LENGTH}}} \\{0,} & {otherwise}\end{matrix} \right.} & (3)\end{matrix}$where length(seg) is the duration of the candidate segment (e.g., inmilliseconds), MAXLENGTH and MINLENGTH are pre-defined target minimumand maximum lengths for a particular level of the hierarchy, andC_(MAXCOST) is a large constant (e.g., 5000). In an exampleimplementation, MAXLENGTH and MINLENGTH are given by:MINLENGTH=500·2^(level)  (4)MAXLENGTH=2·MINLENGTH  (5)For example, for level 2, MINLENGTH=500*4=2000 milliseconds, andMAXLENGTH=4000 milliseconds. In an example embodiment, hierarchicalclustering component 170 computes the length cut cost for a candidatesegmentation as the sum of the length cut costs for each of thecandidate segments in the candidate segmentation. As such, in someembodiments, the length cut cost penalizes segmentations with candidatessegments with durations outside a pre-defined target range.

In some embodiments, hierarchical clustering component 170 computes aboundary cut cost for a candidate segment based on a weighed sum of theboundary cut costs (e.g., silence, sentence, scene, tool, custom, and/orother cut costs) for each of the boundaries associated with thecandidate segment. Depending on the embodiment and/or the type ofboundary cut cost, the boundaries associated with a candidate segmentused to compute the boundary cut cost are the end points of thecandidate segment, the clip atom boundaries enclosed by the candidatesegment, and/or both. In some implementations, the boundary cut costsfor a candidate segment (e.g., silence, sentence and scene cut costs)are weighted to favor certain types of boundaries, such as (e.g.,sentence or scene) boundaries that are adjacent to a long silence. Insome cases, scene boundaries are favored since, in certain contexts,scene boundaries usually signal a relatively larger semantic shift inthe video content. In an example implementation that weights differenttypes of boundary cut costs for different types of boundaries, theboundary cut cost for a candidate segment is given by:cost_(boundary)=cost_(silence)+cost_(sentence)+3*cost_(scene)  (6)where cost_(silence) is the sum of silence cut costs, cost_(sentence) isthe sum of sentence cut costs, and cost_(scene) is the sum of scene cutcosts assigned to the boundaries associated with the candidate segment(e.g., the boundaries at the beginning and end of the candidatesegment), as described in more detail with respect to cut costcomputation component 166 above. In some cases, a boundary has twosilence cut costs (e.g., corresponding to adjacent silence durations,preceding and following the boundary), in which case, someimplementations sum both silence cut costs for each boundary. In someembodiments, hierarchical clustering component 170 computes the boundarycut cost for a candidate segmentation as the sum of the boundary cutcosts computed for each of its candidate segments. Additionally oralternatively, hierarchical clustering component 170 computes theboundary cut cost for a candidate segmentation as the sum of theboundary cut costs assigned to each boundary in the candidatesegmentation.

As explained in more detail above, in some cases, silence cut cost isdependent on the hierarchy level. As such, some implementations computesilence cut cost based on silence duration dur_(silence), normalized bya pre-defined target MAXLENGTH of video segments, which in someembodiments is specific each level. Incorporating an examplelevel-dependent silence cut cost into equation (6), an example boundarycut cost for a candidate segment at a particular hierarchy level isgiven by:cost_(boundary)=(1.0−dur_(silence)/MAXLENGTH)+cost_(sentence)+3*cost_(scene)  (7)As with equation (6), in some embodiments in which each boundary has twoassociated silence cut costs (e.g., corresponding to adjacent silencedurations, preceding and following the boundary), equation (7) isupdated to include (e.g., sum) silence cut costs for each boundary.

In some implementations where other categories of boundaries areadditionally or alternatively defined, a corresponding cut cost isincluded and/or weighted, for example, according to how important thattype of boundary is (e.g., based on the context). For example, in someimplementations with tool boundaries, tool cut cost is included in theboundary cut cost for a candidate segment, such as:cost_(boundary)=cost_(silence)+cost_(sentence)+3*cost_(scene)+0.5*cost_(tool)  (8)

In some embodiments, hierarchical clustering component 170 computes theconsistency cost of a candidate segment based on the sum of the scenecut costs for all clip atoms boundaries that fall within the candidatesegment (e.g., excluding the clip atom boundaries at the start and endpoints of the candidate segment). This consistency cost effectivelypenalizes candidate segments that contain scene boundaries in somewherein the middle of the candidate segment. For example, in animplementation, consistency cost for a candidate segment is given by:cost_(consistency)(seg)=Σ_(b∈seg)1−cost_(scene)(b)  (9)where cost_(scene)(b) is the scene cut cost for clip atom boundary b,and b∈S is all clip atom boundaries in candidate segment S, excludingthe start and end boundaries of S.

Thus, in some embodiments, hierarchical clustering component 170computes multiple cut costs for a candidate segment, and combines themultiple cut costs to compute a total cut cost for the candidatesegment. In an example embodiment, hierarchical clustering component 170computes cut cost for a candidate video segment as:cost_(clip)=(1+cost_(length)+cost_(boundary)+cost_(consistency))²  (10)where cost_(length), cost_(boundary), and cost_(consistency) are thelength cut cost, boundary cut cost, and consistency cut cost for acandidate segment described above. Thus, in some embodiments,hierarchical clustering component 170 computes cut costs for eachcandidate segment in a segmentation, and the cost function for acandidate segmentation sums the cut costs for its candidate segments.Additionally or alternatively, the cost function for a candidatesegmentation sums some or all cut costs for boundaries associated withthe candidate segmentation. These are just a few examples of possiblecost functions, and other variations are contemplated within the scopeof the present disclosure.

As such, in some embodiments, hierarchical clustering component 170 usesa segmentation algorithm (e.g., a line breaking algorithm such as Knuthand Plass' line breaking algorithm, a dynamic programming algorithm)that incorporates any suitable cost function to compute an optimal videosegmentation at a particular hierarchy level (e.g., level 2 and up). Inan example implementation, the input into the segmentation algorithm fora particular hierarchy level is the segmentation from the previous levelof the hierarchy, and the segmentation algorithm determines an optimalsegmentation for the level based on the cost function (e.g., for thelevel). In some embodiments, a segmentation at a given hierarchy levelis represented by a list of IDs and/or time values associated with (i)clip atom boundaries that define the segmentation, (ii) clusters of clipatoms that form the video segments for the segmentation, and/or (iii)the video segments for the segmentation. In some implementations,hierarchical clustering component 170 iteratively applies thesegmentation algorithm to compute segmentations for successive levels ofthe hierarchy, for example, until the number of video segments in ahierarchy level is smaller than some number (e.g., 10), until thesegmentation algorithm returns a single chunk for a hierarchy level,and/or other criteria. As such, in various embodiments, hierarchicalclustering component 170 computes a hierarchical segmentation with aplurality of levels, where each successive level segments the video intovideo segments with an increasing (or decreasing) amount of granularity.

In various embodiments, hierarchical clustering component 170 generatesa representation of the hierarchical segmentation using one or more datastructures. In an example implementation, the hierarchical segmentationis represented using a two dimension array, where the dimensions of thearray correspond to the different levels of the hierarchy, and thevalues stored in each dimension of the array represent video segments ina corresponding hierarchy level (e.g., time ranges and/or anidentification of hierarchical clusters 196 of clip atoms 194 thatdefine the video segments). For example, in some embodiments, levels[0]represents the video segments in the coarsest level of the hierarchy,and levels[levels.length−1] represents the video segments in the finestlevel. In some cases, video segments are represented by valuesrepresenting, or references to, timeline locations (e.g., startTimeand/or endTime, for example, in milliseconds), clip atoms (e.g., IDs),clip atom boundaries (e.g., IDs), and/or other representations. In somecases, a single copy of the video and a representation of boundarylocations are maintained (e.g., as one or more video files 192 andsegment boundaries 198 in storage 190). In other cases, separate copiesof video (e.g., video files 192) and/or separate copies of the videosegments (e.g., chunks of video files 192, such as clip atoms 194 and/orhierarchical clusters 196) are maintained for each level of thehierarchy. Generally, embodiments that maintain separate copies providefor faster access, scrubbing, trimming, and/or the like.

In some embodiments, video ingestion tool 155 includes metadataextraction component 175, which extracts metadata about a video. Forexample, in various embodiments, metadata extraction component 175transcribes audio associated with a video (e.g., using any known audiotranscription technique), visually extracts content or action tags fromvideo frames of the video (e.g., by performing object detection, forexample, using one or more neural networks), extracts software logevents from a temporal log associated with the video (e.g., a softwareusage log, such as one generated while screen capturing orscreencasting), and/or others. In some cases, transcribed audio isstored or otherwise associated with a corresponding video segment (e.g.,in a data structure representing the hierarchical segmentation).Additionally or alternatively, transcribed audio is analyzed for termfrequency, and some or all terms (e.g., the most frequent n terms) arestored as searchable metadata tags associated with corresponding videosegments. As such, in some embodiments, metadata extraction component175 extracts video features such as transcription text, keywords from anaudio transcript, visually extracted content or action tags, and/oraction tags corresponding to extracted log events (e.g., software toolevents), and stores or otherwise associates the extracted video featureswith corresponding video segments. For example, in some embodiments,metadata extraction component 175 includes transcription text, keywords,visually extracted content or action tags, and/or action tagscorresponding to extracted log event tags (or some representationthereof such as an ID or reference) in a representation of thehierarchical segmentation (e.g., a 2D array). As such, the extractedvideo features can be used as searchable metadata tags. Generally,extracting video features from video segments and using the extractedfeatures as searchable metadata tags makes selecting and browsing videosegments easier, as explained in more detail below.

Interacting with Hierarchical Clusters of Video

The prior section described an example technique for ingesting a video,for example, to prepare for video editing or other video interactions.By segmenting the video at semantically meaningful locations,hierarchically clustering the resulting semantic video segments to forma hierarchical segmentation, and/or generating searchable metadata tagsabout the hierarchical clusters, video ingestion tool 155 generates astructured representation of the video that provides an efficient andintuitive structure for interacting with the video, for example, viavideo interaction engine 120 of video editing application 115 in FIG. 1.

In the example illustrated in FIG. 1 , video interaction engine 120provides a user interface that includes one or more interaction elementsproviding various interaction modalities for selecting, navigating,playing, and/or editing a video based on a hierarchical segmentation ofthe video. In FIG. 1 , video interaction engine 120 includes varioustools, such as segmented timeline tool 125, zoom/scroll bar tool 130,metadata panel tool 135, search tool 140, video edit tool 145, and videoplayback tool 148. In various embodiments, these tools are implementedusing code that causes a presentation of a corresponding interactionelement(s), and detects and interprets inputs interacting with theinteraction element(s). For example, segmented timeline tool 125controls a video timeline segmented by the boundaries of thehierarchical clusters, zoom/scroll bar tool 130 controls a zoom/scrollbar that zooms in and out and scrolls across the hierarchical clusterspresented on the video timeline, metadata panel tool 135 controls ametadata panel showing extracted metadata such as transcribed audio andextracted metadata tags for each hierarchical cluster, search tool 140controls a search bar and corresponding search functionality forsearching extracted metadata associated with the hierarchical clusters,video edit tool 145 performs an editing operation on selectedhierarchical clusters of video segments, and video playback tool 148play back selected hierarchical clusters of video segments. Thefunctionality of these and other example video interaction tools isdescribed in more detail below with respect FIGS. 4-7 .

Turning now to FIG. 4 , FIG. 4 illustrates an example user interface 400for interacting with hierarchical clusters of video segments, inaccordance with embodiments of the present invention. In someembodiments, user interface 400 is generated by video interaction engine120 of FIG. 1 . In FIG. 4 , user interface 400 includes video playbackpanel 410, video playback sizing control 420, clip detail control 430,playback cursors 440 and 450, video timeline window 460, zoom/scroll bar470, and metadata panel 480.

In an example use case, a user loads a video for editing, for example,using a file explorer to identify the location of the video (notdepicted). In some cases, upon receiving a command to load the video,the video is ingested to generate a hierarchical segmentation of thevideo (if not previously ingested), and the hierarchical segmentation isloaded. Generally, the total length of video content corresponds to thetotal length of a corresponding timeline for the video, and the videotimeline is segmented according to the hierarchical segmentation. Videotimeline window 460 presents a view of the video timeline, and morespecifically, a view of a particular level of the hierarchicalsegmentation of the video timeline. In some embodiments, video timelinewindow 460 displays a portion of the video timeline with lines, tickmarks, transitions, or some other indication of the boundaries of videosegments of a particular level of the hierarchical segmentation. In somecases, a particular level is loaded by default (e.g., a pre-determinedlevel, a least granular level, a most granular level, a level with videosegments that do not exceed a threshold duration, a level with anaverage video segment duration that does not exceed a thresholdduration, etc.). In some embodiments, an interaction element such asclip detail control 430 is used to navigate and change the level of thehierarchy viewed on video timeline window 460. As such, video timelinewindow 460 displays a view of a selected level of the hierarchicalsegmentation of the video.

In some embodiments, metadata panel 480 presents metadata about videosegments of the hierarchical segmentation, such as transcribed audio,keywords, extracted visual tags, extracted log event tags, and/or thelike. In some embodiments, metadata panel 480 presents a (scrollable)composite list of extracted metadata for all video segments, andsegments the composite list into corresponding metadata segments basedon a selected level of the hierarchical segmentation. In someembodiments, an interaction element such as clip detail control 430 isused to navigate and change the level of the hierarchy used to segmentthe composite list of extracted metadata presented in metadata panel480.

Continuing with a high-level overview, once a video is loaded, videoplayback panel 410 presents a video frame corresponding to a selectedlocation on the presented video timeline and/or a current location ofplayback cursor 440. In some embodiments, a user can select a portion ofthe video timeline presented in video timeline window 460 (e.g., byclicking or tapping on a corresponding portion of the presented videotimeline, dragging playback cursor 440, etc.) to cause presentation of acorresponding video frame in video playback panel 410. Additionally oralternatively, selecting a portion of the video timeline causes the viewof the composite list of extracted metadata in metadata panel 480 tojump to a corresponding metadata segment in the composite list. In someembodiments, clicking or tapping a playback button or other controlelement causes a linear playback of the video in the video playbackpanel 410, playback cursor 440 advances along the video timelinepresented in video timeline window 460 as video playback advances,and/or the composite list of metadata presented in metadata panel 480advances (e.g., automatically scrolls) to a metadata segmentcorresponding to the video segment being played as video playbackadvances.

In some embodiments, video timeline window 460 and/or zoom/scroll bar470 allow a user to select one or more video segments defined by ahierarchical segmentation. As explained above, in some embodiments, thefinest granularity of a hierarchical segmentation defines a set of videosegments (also called clip atoms), and coarser levels of the hierarchydefine video segments that can be thought of clusters video segmentsfrom finer levels. Although some embodiments refer to video segments, itshould be understood that, in some cases, a video segment at aparticular level of hierarchical segmentation is a cluster of videosegments from a finer level. Thus, in some embodiments, interactionswith video segments should be understood to include interactions withhierarchical clusters of (more granular) video segments.

In some embodiments, an interaction element such as clip detail control430 of FIG. 4 is provided to allow the user to navigate to a differentlevel of a hierarchical segmentation of a video timeline. In the examplein FIG. 4 , clip detail control 430 includes two buttons, one thattransitions to a lower level (e.g., finer video segments) and one thattransitions to a higher level (e.g., coarser video segments). Othernon-limiting examples of possible interaction elements for controllingthe level of the hierarchy include a slider, scroll bar, dial, drop-downmenu, input field, and/or others. In another example, the level of thehierarchy is adjusted level based on the zoom level (e.g., as the userzooms in, the hierarchy level becomes finer). In various embodiments,when the user navigates to a different hierarchy level, the videotimeline, the selection of video segments displayed on the videotimeline window, and/or the composite list of metadata segmentspresented in a metadata panel are updated to reflect the boundaries ofthe selected hierarchy level, allowing for a refined selection of aportion of the video through selection of video segments and/or metadatasegments with different levels of granularity.

FIGS. 5A-5I are illustrations of example interactions with hierarchicalclusters of video segments using a video timeline window and/or azoom/scroll bar, in accordance with embodiments of the presentinvention. For example, FIG. 5A shows an embodiment with video timelinewindow 510 segmented according to a particular level of a hierarchicalsegmentation, playback cursor 505 showing a current playback position ofthe video, and zoom/scroll bar 520 (described in more detail below).FIG. 5A also shows selection cursor 501 prior to the user making aselection. In this example, selection cursor 501 is illustrated as anarrow icon, indicating the user has not clicked or otherwise usedselection cursor 501 to initiate a selection. In this embodiment, videosegment 530 is outlined but not shaded, indicating selection cursor 501is hovered over video segment 530 without selecting video segment 530.Various ways of emphasizing different selection states are described inmore detail below.

In some embodiments, when a user selects a video segment presented onvideo timeline window 510 (e.g., by clicking or tapping a portion of thetimeline between two boundaries), the video segment is selected and theselection snaps to the boundaries of the video segment. Variousembodiments support selection of multiple video segments, for example,using a click (or tap and hold) and drag operation, by toggling amultiple selection (e.g., control+multiple clicks), and/or other usingother any suitable technique. In some embodiments, when the user selectsa video segment (e.g., using a click, a drag operation, and/orotherwise), the first frame of the selected video segment is displayedin video playback panel 410 (e.g., as opposed to the frame in the middleof the segment corresponding to position on the timeline where the userclicked).

In some embodiments, video timeline window 510 supports a click and dragoperation (or tap, hold, and drag operation) to select multiple videosegments. In an example embodiment, an initial click (or tap) on a videosegment that is not part of an existing selection serves to select andemphasize (e.g. highlight) the video segment. By clicking and draggingfrom a selected segment to an unselected segment, the selection isexpanded to include the unselected segment. As the drag operationcrosses the boundary between the selected and unselected segment, theselection expands, snapping to the segment boundary of the previouslyunselected segment. On the other hand, reversing direction and draggingfrom an outer or most recently selected segment to an inner orpreviously selected segment reduces the selection by deselecting theouter or most recently selected segment. As the drag operation crossesthe boundary between an outer segment in the selection and an innersegment in the selection, the selection shrinks by snapping to theboundary between the inner and outer segments (de-selecting the outersegment). As such, if the user clicks and drags across multiple videosegments on video timeline window 510, the drag operation adds videosegments to the selection (e.g., as the drag operation expands theselection) or removes video segments from a selection (e.g., as the dragoperation reduces the selection). Thus, a user can drag across a videotimeline to make a selection that snaps to video segment (e.g., cluster)boundaries.

FIGS. 5B-5C illustrate an example click and drag operation. For example,in FIG. 5B, the user clicks on a video segment and holds, which selectsthe video segment, displays selection 540 emphasizing (e.g.,highlighting) the selected video segment, and changes selection cursor501 from an arrow icon in FIG. 5A to a selection icon in FIG. 5B. (Insome embodiments, the icon used for selection cursor 501 depends onwhether the user is making an active selection.) In FIG. 2C, the usercontinues the drag operation by dragging selection cursor 501 to theright along video timeline 510. As the user clicks and drags, whenselection cursor 501 crosses a boundary of the hierarchicalsegmentation, selection 540 expands by snapping to the subsequentboundary for the selected segment. In the example illustrated in FIG.5C, the boundary of selection 540 that is being moved (e.g., boundary550) is emphasized in a different way than the other boundary ofselection 540 that is not being moved. For example, boundary 550 isillustrated as a handle with two end points. As the drag operationcrosses a segment boundary, the handle snaps the next segment boundary.

FIG. 5D illustrates multiple selections 560 and 565 of disjoint videosegments. For example, a user might first click on a video segmenthighlighted by selection 560, input some command toggling a multipleselection (e.g., by holding the control key), and then initiating aclick and drag operation to add selection 565 to the multiple selection.In this example, left boundary 567 of selection 565 is illustrated as ahandle, indicating left boundary 567 of selection 565 is the selectionboundary being modified (e.g., as the drag operation moves to the left).In some cases when there are multiple disjoint selections, when a dragoperation defining one selection crosses into a second selection (oradvances to the point that the first selection overlaps with the secondselection), the two selections are collapsed into one compositeselection. In some embodiments, the original disjoint selections aremaintained such that reversing the drag operation removes the overlapand reinstates the original disjoint selections.

In another example, dragging across an initial click location in a dragoperation and into an adjacent segment de-selects the initially clickedsegment and selects the adjacent segment. In other words, in thisexample, if a user clicks a first segment (which selects the firstsegment) and drags right, but then changes direction and drags to theleft of the first segment, the first segment is de-selected, and thesegment to the left of the first segment is selected. As such, the dragoperation can continue along either direction of the video timeline,snapping to segment boundaries while dragging along the timeline.

In some embodiments, a zoom bar and/or a scroll bar (such as zoom/scrollbar 520) controls the view of the video timeline presented in videotimeline window 510. For example, in in the embodiment a zoom/scroll barincludes a thumb (or bar) that can be dragged along a track (or trough).In some cases, the thumb has independently moveable (e.g., draggable)endpoints that control a corresponding location on the video timelinepresented in the window. Thus, in some embodiments, resizing the thumbzooms in and out of the timeline window, and/or dragging the thumb alongthe track scrolls the video timeline through video timeline window 510.

For example, continuing with the example illustrated in FIG. 5D, assumethe user expands selection 565 to include selection 560, and the twoselections are collapsed to form selection 568, as illustrated in FIG.5E. FIGS. 5E-5G depict examples of zooming and scrolling across videotimeline window 510 (and selection 568) using zoom/scroll bar 520. InFIG. 5E, zoom/scroll bar 520 is illustrated with a thumb havingendpoints 570 and 575. In this example, the thumb takes up the entiretrack of zoom/scroll bar 520, so video timeline window 510 displays theentire video timeline, including selection 568. Assume the user movesselection cursor 501 over endpoint 570 of the thumb, as illustrated inFIG. 5E, and moves endpoint 570 from the position illustrated in FIG. 5Eto the position illustrated in FIG. 5F. By shrinking the size of thethumb on zoom/scroll bar 520, the user zooms into the video timeline onvideo timeline window 510, illustrated in FIG. 5F. Accordingly, in FIG.5F, the appearance of the video segments on video timeline window 510and of selection 568 grows in size as the user zooms into the videotimeline (although selection 568 is not expanded to include anyadditional video segments).

Assume now that the user further adjusts the locations of endpoints 570and 575 from the locations illustrated in FIG. 5F to the locationsillustrated in FIG. 5G. Accordingly, FIG. 5G illustrates a correspondingzoomed in view of the video timeline and selection 568 in video timelinewindow 510. In this case, selection 568 has been zoomed in by an amountthat its right boundary is no longer visible on video timeline window510. In this example, thumb 580 is selectable and can be dragged alongthe track of zoom/scroll bar 520, which serves to scroll the videotimeline across video timeline window 510. As such, to view the rightboundary of selection 568, the user can grab thumb 580 and drag, forexample, to the right.

FIGS. 5H-5I illustrate another example embodiment that involveszooming/scrolling. For example, FIG. 5H includes video timeline window511 showing a portion of the video timeline that includes playbackcursor 506 and a selection of two video segments. FIG. 5H also includeszoom/scroll bar 521 with playback cursor 590. In this example, playbackcursor 506 of video timeline window 511 shows the playback position ofthe video on the video timeline. Since the view of the video timeline invideo timeline window 511 can be zoomed in and out and scrolled left orright, the location of playback cursor 506 indicating a stationaryplayback position (e.g., paused video) moves through video timelinewindow 511, depending on how the video timeline is zoomed or scrolled.By contrast, the position of playback cursor 590 on zoom/scroll bar 521shows the relative location of the playback position with respect to theentire timeline. For example, when video timeline window 511 is zoomedout all the way so the entire video timeline is displayed, the positionsof playback cursor 506 and playback cursor 590 track one another, sodragging one cursor has the effect of moving the other cursor the samedistance. In FIG. 5H, the user selects playback cursor 506 (causing themouse icon to change and selection effect 595 to be displayed), anddrags to the left or right, which causes both playback cursor 506 andplayback cursor 590 to move in synch.

Now assume the user resizes the thumb of zoom/scroll bar 520 byrepositioning its endpoints. In some embodiments, when the user grabs onendpoint 570 (causing the mouse icon to change and selection effect 597to be displayed, as illustrate in FIG. 5I) and moves endpoint 570 fromthe position illustrated in FIG. 5H to the position illustrated in FIG.5I, the view of the video timeline presented in video timeline window511 zooms in. Similarly, the view of the selected video segments zoomsin. In this example, since the view of the video timeline is changing(zooming), the relative position of playback cursor 506 on videotimeline window 511 moves as zooming changes the view (although theposition of playback cursor 506 on the video timeline itself does notchange).

By contrast, in this example, zooming into the video timeline (orscrolling the view across the video timeline) does not change theposition of playback cursor 590 on zoom/scroll bar 521 because, in thisexample, the playback position does not change (e.g., because the videois paused). As such, the relative location of the playback position withrespect to the entire timeline does not change, so the position ofplayback cursor 590 on zoom/scroll bar 521 does not change. In otherwords, in this embodiment, using zoom/scroll bar 521 to change the viewof the video timeline on video timeline window 511 (e.g., whether movingthe endpoints of the thumb or dragging the thumb across the track)changes the position of playback cursor 506 in the video timeline window511, but not the position of playback cursor 590 on zoom/scroll bar 521.As such, in this example, playback cursor 506 on video timeline window511 shows playback position with an adjustable time scale, whileplayback cursor 590 on zoom/scroll bar 520 shows playback position witha fixed time scale. Presenting multiple indications of playback positionat different time scales (e.g., on parallel spectra) provides a simpleway of presenting multiple perspectives, enhancing the user's ability tovisualize and comprehend the video timeline. It should be understoodthat this is just an example, and other ways of presenting multiple timescales whether fixed or adjustable are possible. For example, in someembodiments, the position of playback cursor 590 is presented relativeto the thumb of zoom/scroll bar 520, rather than (or in addition to)being presented relative to the entire track.

Turning now to FIGS. 6A-6J, FIGS. 6A-6J illustrate example userinterfaces for interacting with hierarchical clusters of video segmentsusing metadata panel 604 and/or a metadata search. Generally, metadatapanel 604 presents metadata (e.g., transcribed audio and extractedmetadata tags) for each video segment. In these examples, metadata panel604 presents a scrollable, composite list of the metadata for all videosegments, and the composite list is segmented into metadata segments atlocations that correspond to the boundaries of the level of thehierarchy being viewed. In some embodiments, each of the metadatasegments is independently selectable, which emphasizes (e.g.,highlights) the selected metadata segment, emphasizes the correspondingvideo segment on the video timeline in video timeline window 602, movesplayback cursors 610 and 612 to locations corresponding to the firstvideo frame of the corresponding video segment, and/or displays thevideo frame in video player 601.

In some embodiments, a search bar (e.g., search bar 640 of FIGS. 6C-6J)is provided for searching metadata. In these embodiments, a user entersone or more keywords in search bar 640, and extracted metadataassociated with the video segments is searched for words that match thekeyword search. Examples of extracted metadata include transcribedaudio, frequent transcript terms, visually extracted content or actiontags, extracted action tags corresponding to extracted software events,and/or other extracted features. In some embodiments, matching videosegments (i.e., segments with matching metadata) are emphasized (e.g.,highlighted) on the video timeline in video timeline window 602, and/orcorresponding matching metadata segments are emphasized (e.g.,highlighted) in metadata panel 604. In some embodiments, when the usernavigates to a different hierarchy level (e.g. using clip detail control660 illustrated in FIGS. 6E-H), the video timeline displayed in videotimeline window 602 and/or metadata panel 604 are updated to reflect theboundaries of the selected hierarchy level, and the search results(matching video segments and/or metadata segments) are updated based onthe boundaries of the selected hierarchy level. Thus, in someembodiments, changing the level of hierarchy during an active search(e.g., with highlighted search results) can transform a set of searchresults into corresponding coarser or finer segments defined by adifferent level of the hierarchy, allowing for a more flexible andefficient search experience.

In various embodiments, different types of emphasis are applied torepresent different selection states for video segments (e.g., clustersof clip atoms) presented in video timeline window 602 and/orcorresponding metadata segments presented in panel 604. For example,some embodiments may apply different types of emphasis to unselectedvideo segments, a video segment corresponding to a current playbackposition, a video or metadata segment being hovered over, clicked orhighlighted video or metadata segments, video or metadata segments withmetadata tags that match a keyword search, video segments (andcorresponding metadata segments) that have been added to a selectionqueue (e.g., a playback queue), some combination thereof, and/or others.Examples of different types of emphasis include different colors,gradients, patterns, outlines, shadows, and/or others. In the examplesillustrated in FIGS. 6A-6J, different selection states are illustratedwith different outline types and/or different greyscale shades.

FIG. 6A illustrates an example user interface with video timeline window602, zoom/scroll bar 603, and metadata panel 604. In this example,metadata panel 604 presents a composite list of metadata segments (e.g.,metadata segments 615 and 625) that include extracted metadata such astranscribed audio visually extracted content or action tags. In thisexample, video segment 620 on video timeline window 602 corresponds tometadata segment 615, which includes a transcription of the voice overfrom the audio track of video segment 620, as well as visually extractedcontent or action tags extracted from video frames of video segment 620(e.g., using one or more neural networks to perform object detection onthe video frames). In this example, video segment 620 is outlined usinga particular line type, indicating playback cursor 610 is located invideo segment 620. Further, video segment 620 and corresponding metadatasegment 615 are outlined using the same line type, reflecting theircorrespondence. As illustrated in FIG. 6A, the video frame in videoplayer 601 corresponding to the position of playback cursor 610 includesa sunset, and corresponding metadata segment 615 includes acorresponding portion of the transcribed audio (illustrated as dummytext) and related content tags, such as outdoor, sunset, sky, sun, andclouds. In the example illustrated in FIG. 6A, metadata segment 625 isoutlined using outlined using a line type (different than the line typefor the outline of metadata segment 615) indicating metadata segment 625is being moused over, but has not been selected.

Turning now to FIG. 6B, assume the user clicks on video segment 630 onvideo timeline window 602 and moves playback cursor 610 to video segment635. In some embodiments, dragging playback cursor 610 (and/or playbackcursor 612) onto a particular video segment automatically scrolls thecomposite list of metadata in metadata panel 604 to the location of ametadata segment corresponding to the position of the playback cursor.Conversely, in some embodiments, clicking on a particular metadatasegment automatically updates the position of playback cursor 610 and/orplayback cursor 612 to location associate with a corresponding videosegment. In order to illustrate the correspondence between video andmetadata segments in FIG. 6B, corresponding segments are labeled withthe same reference number and outlined using the same line type. Thus,video segment 635 in video timeline window 602 and correspondingmetadata segment 635 in metadata panel 604 are outlined using a firstline type indicating playback cursor 610 is located in video segment635. Furthermore, video segment 630 in video timeline window 602 andcorresponding metadata segment 630 in metadata panel 604 are outlinedusing a second line type indicating the user has clicked on one of thesegments.

FIG. 6C illustrates an example keyword search of video metadata. In FIG.6C, a user navigates to search bar 640 and starts typing. In someembodiments, when a user enters search bar 640 and/or starts typing,pop-up 645 is presented with keyword suggestions, for example,corresponding to the most frequent n keywords and/or metadata tags forthe video. Depending the implementation, any suitable metric is used toidentify top keywords or metadata tags (e.g., total number of keywordoccurrences in the entire transcript, counting any number of occurrencesof keywords in a particular metadata segment once, counting metadatatags once for each video frame from which a metadata tag was extracted,counting metadata tags once for each metadata segment that includes atag, counting occurrences based on metadata segments corresponding to aparticular hierarchy level such as an active level being viewed, and/orother examples).

Continuing with the example illustrated in FIG. 6C, assume the usertypes in the keyword “bird” into search bar 640 and executes the search.In this example, the metadata for the video (e.g., the composite listpresented in metadata panel 604 and/or the transcript) is searched formatches. FIG. 6C illustrates an example in which video segments 650 havemetadata that matches the keyword search. Note that in FIGS. 6A-6J, themetadata segments are illustrated using dummy text to representtranscribed audio, so matching keywords are not illustrated. In someembodiments, upon detecting a match, video segments 650 and/orcorresponding metadata segments in metadata panel 604 are emphasized toindicate the match. In some cases, video segments 650 are outlined witha particular line type indicating video segments with metadata thatmatches a keyword search. In the example illustrated in FIG. 6C, allsearch results are emphasized the same way, but this need not be thecase. For example, in some embodiments where the transcript is not partof the metadata, results from metadata-keyword matches are emphasizeddifferently (e.g., in a different color) than results fromtranscript-keyword matches. In some embodiments, upon detecting matchingvideo segments, the matching video segments are animated on videotimeline window 602 (e.g., with a jitter or oscillation). In some cases,a transient jitter or oscillation is induced on matching video segments,where the movement is in a direction perpendicular or parallel to thedirection of the video timeline. In some embodiments, the animations ofmatching video segments are synchronized, whether in phase or out ofphase. For example, in some embodiments, a transient jitter oroscillation is induced on successive matching video segments at aparticular interval, giving the appearance of a traveling wave down thevideo timeline, affecting only the matching video segments. In FIG. 6C,matching video segments 650 are illustrated with a slight displacementrelative to one another to illustrate an example traveling wave effect.

Turning now to FIG. 6D, assume the user drags playback cursor 610 overto matching video segments 650. In response, metadata panel 604 scrollsto display corresponding metadata segments (e.g., metadata segment 655).In some embodiments, different emphasis is applied to a matching segmentwhen playback cursor 610 is located within the matching segment, asillustrated by the two different video segments of matching videosegments 650 in FIG. 6D.

Assume now the user wants to take a closer look at the matching videosegments. Accordingly, the user resizes the view of the video timelinein video timeline window 602 using zoom/scroll bar 603 to zoom intomatching video segments 650, as illustrated in FIG. 6E. Note that inthis example, matching video segments 650 are illustrated with differentemphasis depending on whether playback cursor 610 is within a matchingvideo segment. Similarly, corresponding matching metadata segments 655are also illustrated with corresponding different emphases.

Until now, the user has been searching and interacting with videosegments corresponding to a particular level of a hierarchicalsegmentation. Assume the user wants to view search results with a finergranularity. As such, the user can click on the right button in clipdetail control 660 to change the level of the hierarchy to display videoand metadata segments with finer granularity. In some cases, navigatingto a finer level of granularity in a hierarchical segmentation isequivalent to displaying smaller (or finer) semantic clusters. In thisexample, changing from a coarser to a finer level of the hierarchychanges boundaries displayed on the video timeline in video timelinewindow 602 (including matching video segments 650) and the metadata inmetadata panel 604 (including matching metadata segments 655) from thelocations illustrated in FIG. 6E to the locations illustrated in FIG.6F. Subsequent clicks of the right button in clip detail control 660change the boundaries from the locations illustrated in FIG. 6F to thelocations illustrated in FIG. 6G to the locations illustrated in FIG.6H. Note that since transcribed audio is represented using dummy text inthese figures, the correspondences between transcribed audio andcorresponding video segments, and between coarser and finer metadatasegments, are not illustrated. In some embodiments, when changinghierarchy levels, an animation (e.g., a jitter or oscillation) isapplied to the matching video segments corresponding to an updatedhierarchy levels.

From FIG. 6E to FIG. 6F to FIG. 6G to FIG. 6H, matching video segments650 and matching metadata segments 655 are split up into smallersegments, enabling the user to view search results and/or define aselection with more precision. For example, in FIG. 6G, notice howmatching video segments 650 form a continuous portion of the videotimeline, whereas in FIG. 6H, matching video segments 650 are shown withfiner clip detail such that they form disjoint clusters of videosegments on the video timeline. In FIG. 6I, the user zooms in byresizing thumb 665 on zoom/scroll bar 603 and clicks on the left mostmatching video segment 650 to pull up a corresponding matching metadatasegment in metadata panel 604. As such, the user can navigate the videotimeline using video timeline window 602 and zoom/scroll bar 603, andnavigate corresponding metadata segments in metadata panel 604, tointeract with semantic video segments with a level of granularitydefined by a corresponding level of a hierarchical segmentation.

Generally, a user may want to place certain video segments into anoperational queue to perform some type of operation on selected videosegments. FIG. 6J illustrates an example in which a user assigns videosegments to an operational queue by clicking on check boxes 680 in acorresponding metadata segment in metadata panel 604. In this example,the user has activated check boxes 680 for metadata segments 685 and688. In this example, metadata segments 685 and corresponding videosegments 690 are emphasized (e.g., outlined using a particular linetype) in a manner that indicates corresponding video segments 690 havebeen added to the operational queue. Similarly, metadata segment 688 andcorresponding video segment 692 are emphasized (e.g., outlined using aparticular line type) in a manner that indicates video segment 692 hasbeen added to the operational queue and video segment 692 and/or hasmetadata segment 688 has been clicked. This example demonstrates ascenario with different selection states for segments that have beenselected by clicking or highlighting versus segments that have beenselected by activating associated activated check boxes 680. However, insome embodiments, selecting multiple segments from video timeline window602 (e.g., using a click and drag operation, by toggling a multipleselection and clicking on multiple segments) assigns the selectedsegments to an operational queue. These are just a few examples, andother selection techniques are possible within the scope of the presentdisclosure.

Turning now to FIG. 7 , FIG. 7 illustrates an example user interface forinteracting with hierarchical clusters of video segments based onsoftware log events, in accordance with embodiments of the presentinvention. Similar to the examples illustrated in FIGS. 6A-6J, FIG. 7illustrates an example user interface with video timeline window 702,zoom/scroll bar 703, metadata panel 704, and search bar 740. Whereas theextracted and searchable metadata represented in FIGS. 6A-6J includedtranscribed audio and visually extracted content or action tags, theextracted and searchable metadata illustrated in FIG. 7 additionallyincludes action tags corresponding to extracted software events (e.g.,software tool events).

In this example, a user has previously recorded a BEHANCE live stream ofhis PHOTOSHOP usage, which generates a software usage log of the user'sactions in PHOTOSHOP. When the recording of the live stream is ingested,the tool events captured in the software usage log are extracted andused to place or otherwise associate searchable action tags (e.g.,corresponding to different tool selections, transitions, and/or uses)with corresponding locations on the video timeline. As such, in someembodiments, the action tags (e.g., the locations of the action tags onthe video timeline) are associated with corresponding video segmentsdefined by a hierarchical segmentation based on their locations in time.Thus, in some embodiments, the action tags are included in or otherwiseassociated with the hierarchical segmentation, enabling the action tagsto be used as searchable metadata tags to identify matching portions ofthe video timeline and/or corresponding matching video segments. Assuch, in this example, metadata segments (e.g., metadata segment 715)are presented in metadata panel 704 with action tags (e.g., action tags717) corresponding to the tool events that take place in a correspondingvideo segment.

Similar to the user interface illustrated in FIG. 6C, when a usernavigates to search bar 740 and/or starts typing in search bar 740,pop-up 745 is presented with keyword suggestions. In this case, keywordsuggestions include separate clusters, cluster 747 of the most frequentn keywords and/or visually extracted tags and cluster 749 of the mostfrequent n software tool event tags. Whether the user selects one of thesuggestions from pop-up 745 or types in keyword and executes the search,the metadata for the video (e.g., the composite list presented inmetadata panel 604) is searched for matches. In some embodiments,clicking on a suggestion from pop-up 745 only searches correspondingmetadata tags (e.g., selecting a software tool event tag only searchesextracted software tool event tags). As such, a metadata search can beused to search any type of extracted metadata and quickly identifymatching video segments.

Depending on the implementation, any number and variety of operationsare performed on selected video segments (and/or a corresponding portionof the video). For example, video segments selected from a videotimeline window (e.g., using a click and drag operation, by toggling amultiple selection and clicking on multiple segments), video segmentscorresponding to metadata segments selected from a metadata panel (e.g.,using a click and drag operation, by toggling a multiple selection andclicking on multiple segments, by checking associated check boxes) areplaced in an operational queue. Depending on the implementation, varioustypes of tools are provided to perform a corresponding operation on thevideo segments in the operational queue. In one example, the videosegments in the operational queue are played back (e.g., upon clicking aplay button), skipping video segments that are not placed in anoperational queue. In another example, the video segments in theoperational queue are trimmed (e.g., by removing the unselected videosegments), edited in some other way (e.g., by rearranging, cropping,applying transitions or effects, adjusting color, adding titles orgraphics), exported, or otherwise. Depending on the implementation, anyknown tool or technique is used to perform any type of operation on thevideo segments in the operational queue.

Example Flow Diagrams

With reference now to FIGS. 8-25 , flow diagrams are providedillustrating various methods. Each block of the methods 800 through 2500and any other methods described herein comprise a computing processperformed using any combination of hardware, firmware, and/or software.For instance, various functions can be carried out by a processorexecuting instructions stored in memory. The methods can also beembodied as computer-usable instructions stored on computer storagemedia. The methods can be provided by a standalone application, aservice or hosted service (standalone or in combination with anotherhosted service), or a plug-in to another product, to name a few.

Turning initially to FIG. 8 , FIG. 8 illustrates a method 800 forgenerating a hierarchical segmentation of a video timeline, inaccordance with embodiments described herein. Initially at block 810, asegmentation of a video timeline of a video is generated by detectingboundaries of clip atoms of unequal durations of the video. At block820, a representation of a hierarchical segmentation of the videotimeline is generated. Each level of the hierarchical segmentationsegments the video timeline into a set of video segments divided by acorresponding set of the boundaries. At block 830, at least one of thelevels of the hierarchical segmentation of the video timeline isprovided for presentation.

Turning now to FIG. 9 , FIG. 9 illustrates a method 900 forhierarchically clustering semantic video segments into a hierarchicalsegmentation, in accordance with embodiments of the present invention.Initially at block 910, boundaries of semantic video segments of a videoare detected from content of the video. At block 920, the semantic videosegments are hierarchically clustered into a hierarchical segmentationof a video timeline of the video. Each level of the hierarchicalsegmentation clusters the semantic video segments with a correspondinglevel of granularity. At block 930, at least one of the levels of thehierarchical segmentation of the video timeline is provided forpresentation.

Turning now to FIG. 10 , FIG. 10 illustrates a method 1000 for detectingboundaries of clip atoms, in accordance with embodiments of the presentinvention. For example, method 1000 illustrates a possible way ofperforming at least a portion of block 810 of FIG. 8 or block 910 ofFIG. 9 . Method 1000 starts with video 1010, which is separated intoaudio component 1020 and visual component 1030 (e.g., an audio track anda video track). Voice activity detection (VAD) is applied to audiocomponent 1020 to generate VAD scores, which are used as VAD cost 1040.Further, audio component 1020 is transcribed using any knownspeech-to-text algorithm to generate transcript 1050. Transcript 1050 issegmented to identify sentence boundaries 1070 and utterance boundaries1060 (e.g., word boundaries). The locations of sentence boundaries 1070and utterance boundaries 1060 in transcript 1050 are mapped to locationson the video timeline of video 1010. On the visual side, visualcomponent 1030 is analyzed to identify shot boundaries 1080 (also calledscene boundaries) by detecting abrupt visual changes, and the locationsof shot boundaries 1080 are mapped to locations on the video timeline ofvideo 1010. In some embodiments, any number of adjustment rules areapplied to move the location of one or more of the boundaries on thevideo timeline, to merge boundaries, to remove boundaries, or apply someother adjustment. Finally, utterance boundaries 1060, sentenceboundaries 1070, and shot boundaries 1080 are used to identify thelocations of boundaries for clip atoms of video 1010. In someembodiments, clip atoms 1090 are simply denoted by a list of theboundaries. Additionally or alternatively, clip atoms 1090 are generatedas separate video clips.

Turning now to FIG. 11 , FIG. 11 illustrates a method 1100 for detectingand adjusting locations of speech boundaries, in accordance withembodiments of the present invention. For example, method 1100illustrates a possible way of performing at least a portion of block 810of FIG. 8 or block 910 of FIG. 9 . Initially at block 1110, speechboundaries are detected from audio of a video. The speech boundariesdefine speech segments and non-speech segments. At block 1120,voice-activity detection (VAD) scores of the audio are determined. Atblock 1130, a temporal buffer is added to the speech segments byapplying smoothing to the VAD scores. At block 1140, speech boundariesare snapped to locations within a neighborhood of each speech boundarywhere the VAD scores are at a minimum. At block 1150, a non-speechsegment between two speech segments is closed by searching thenon-speech segment for a minimum VAD score and merging boundariessurrounding the non-speech segment into an adjusted boundary at alocation of the minimum VAD score

Turning now to FIG. 12 , FIG. 12 illustrates a method 1200 for snappingspeech boundaries to proximate scene boundaries, in accordance withembodiments of the present invention. For example, method 1200illustrates a possible way of performing at least a portion of block 810of FIG. 8 or block 910 of FIG. 9 . Initially at block 1210, speechboundaries are detected from audio of a video. The speech boundariesdefine speech segments and non-speech segments. At block 1220, sceneboundaries are detected from video frames of the video. At block 1230,one of the scene boundaries that falls within one of the non-speechsegments is identified. At block 1240, a proximate speech boundary, ofthe speech boundaries, that is within a neighborhood of the sceneboundary is identified. At block 1250, the proximate speech boundary issnapped to the scene boundary.

Turning now to FIG. 13 , FIG. 13 illustrates a method 1300 forextracting event boundaries of log events from a temporal log, inaccordance with embodiments of the present invention. For example,method 1300 illustrates a possible way of performing at least a portionof block 810 of FIG. 8 or block 910 of FIG. 9 . Initially at block 1310,a first set of boundaries are detected from content of the video. Atblock 1320, a second set of boundaries are detected by extracting eventboundaries of log events from a temporal log associated with the video.In some embodiments, block 1320 is performed using the steps illustratedin blocks 1330-1350. At block 1330, times of the log events areextracted from the temporal log. At block 1340, the times of the logevents are mapped to locations on a video timeline. At block 1350, theevent boundaries are associated with the locations on the videotimeline.

Turning now to FIG. 14 , FIG. 14 illustrates a method 1400 for formingdifferent levels of a hierarchical segmentation, in accordance withembodiments of the present invention. For example, method 1400illustrates a possible way of performing at least a portion of block 820of FIG. 8 or block 920 of FIG. 9 . Initially at block 1410, a firstlevel of the hierarchical segmentation is formed with the clip atoms. Atblock 1420, a second level of the hierarchical segmentation is formed bymerging non-speech clip atoms having a duration below a threshold andremoving speech boundaries that occur within a sentence. At block 1430,a third level of the hierarchical segmentation is formed by evaluating acost function for candidate sets of the boundaries sampled from a priorlevel of the hierarchical segmentation to identify an optimalsegmentation of the video timeline for the third level.

Turning now to FIG. 15 , FIG. 15 illustrates a method 1500 for selectinga video segment defined by a hierarchical segmentation, in accordancewith embodiments of the present invention. Initially at block 1510, apresentation of a first level of a hierarchical segmentation of a videois caused. Each level of the hierarchical segmentation is associatedwith a corresponding set of video segments divided by boundaries ofclusters of clip atoms of unequal durations of the video. At block 1520,in response to an input selecting a first video segment defined by thefirst level of the hierarchical segmentation, an update to a selectionstate of the first video segment and a presentation of a visualindication of the selection state on the presentation of the first levelof the hierarchical segmentation are caused. At block 1530, an operationis executed on the first video segment.

Turning now to FIG. 16 , FIG. 16 illustrates a method 1600 for executingan operation on an identified cluster defined by a hierarchicalsegmentation, in accordance with embodiments of the present invention.Initially at block 1610, a presentation of a first level of ahierarchical segmentation of a video timeline of a video is caused. Eachlevel of the hierarchical segmentation clusters semantic video segmentsof the video with a corresponding degree of granularity. At block 1620,in response to an input identifying a first cluster of the semanticvideo segments defined by the first level of the hierarchicalsegmentation, an update to a selection state of the first cluster and anupdate to the presentation of the first level of the hierarchicalsegmentation to include a visual indication of the selection state ofthe first cluster are caused. At block 1630, an operation is executed onthe first cluster.

Turning now to FIG. 17 , FIG. 17 illustrates a method 1700 for updatinga presentation of a first level of the hierarchical segmentation inresponse to navigating to a different level, in accordance withembodiments of the present invention. In some embodiments, method 1700is performed after the steps illustrated in FIG. 15 or FIG. 16 .Initially at block 1710, an input navigating from a first level to adifferent level of a hierarchical segmentation is detected. At block1720, a presentation of the first level is updated to reflect thedifferent level of the hierarchical segmentation. At block 1730, a setof second video segments, defined by the different level of thehierarchical segmentation, and that corresponds to a selected firstvideo segment defined by the first level is identified. At block 1740, apresentation of a visual indication of a selection state, of the secondvideo segments in the set, on the presentation of the different level ofthe hierarchical segmentation is caused.

Turning now to FIG. 18 , FIG. 18 illustrates a method 1800 for snappinga selection to boundaries of clusters of clip atoms, in accordance withembodiments of the present invention. Initially at block 1810, apresentation of a level of a hierarchical segmentation of a videotimeline of a video is caused. Each level of the hierarchicalsegmentation segments the video timeline into video segments divided byboundaries of clusters of clip atoms of unequal durations of the video.At block 1820, in response to an input interacting with a portion of thevideo timeline corresponding to a first video segment defined by thelevel, a selection is snapped to a set of the boundaries correspondingto the input and the level. At block 1830, an operation is performed onthe first video segment.

Turning now to FIG. 19 , FIG. 19 illustrates a method 1900 for snappinga selection to boundaries of clusters semantic video segments, inaccordance with embodiments of the present invention. Initially at block1910, a hierarchical segmentation of a video timeline of a video isaccessed. Each level of the hierarchical segmentation clusteringsemantic video segments of the video with a corresponding level ofgranularity. At block 1920, a presentation of the video timeline beingsegmented by boundaries, of clusters of the semantic video segments,associated with a first level of the hierarchical segmentation iscaused. At block 1930, in response to an input interacting with a firstof the clusters on the video timeline, a selection is snapped to a firstset of the boundaries of the first level corresponding to the input.

Turning now to FIG. 20 , FIG. 20 illustrates a method 2000 for selectingvideo segments using a drag operation, in accordance with embodiments ofthe present invention. For example, method 2000 illustrates a possibleway of performing at least a portion of block 1820 of FIG. 18 or block1930 of FIG. 19 . Initially at block 2010, initiation of a dragoperation is detected at an initial location on the video timelinebetween a first segment boundary and a second segment boundary of thefirst video segment. In response, at block 2020, a selection is definedto include the first video segment by snapping a first selectionboundary of the selection to the first segment boundary and a secondselection boundary of the selection to the second segment boundary. Atblock 2030, the drag operation is detected crossing the second boundaryinto a second video segment defined by the second segment boundary and athird segment boundary. In response, at block 2040, the selection isupdated to include the second video segment by snapping the secondselection boundary from the second segment boundary to the third segmentboundary. At block 2050, the drag operation is detected re-crossing thesecond segment boundary from the second video segment to the first videosegment. In response, at block 2060, the selection is updated to removethe second video segment by snapping the second selection boundary fromthe second segment boundary to the third segment boundary. At block2070, the drag operation is detected crossing the initial location ofthe drag operation. At block 2080, the drag operation is detectedcrossing the first segment boundary from the first video segment into athird video segment defined by the first segment boundary and a fourthsegment boundary. In response, at block 2090, the selection is updatedto remove the first video segment and include the third video segment bysnapping the second selection boundary from the second segment boundaryto the fourth boundary.

Turning now to FIG. 21 , FIG. 21 illustrates a method 2100 foremphasizing a video segment in response to an input identifying aselectable metadata segment, in accordance with embodiments of thepresent invention. Initially at block 2110, a presentation of acomposite list of metadata of a video is caused. The composite list issegmented into selectable metadata segments at locations in thecomposite list corresponding to boundaries of video segments defined bya level of a hierarchical segmentation of a video timeline of the video.At block 2120, an input is detected identifying one of the selectablemetadata segments defined by the level. In response, the stepsillustrated in blocks 2130-2050 are performed. At block 2130, acorresponding one of the video segments is caused to be emphasized onthe video timeline. At block 2140, a movement of a cursor to a firstvideo frame of the corresponding video segment is caused. At block 2150,a presentation of the first video frame is caused.

Turning now to FIG. 22 , FIG. 22 illustrates a method 2200 for updatinga video timeline in response to an input identifying a metadata segment,in accordance with embodiments of the present invention. Initially atblock 2210, a presentation of a composite list of metadata of a video iscaused. The composite list is segmented into metadata segments atlocations in the composite list corresponding to boundaries of videosegments defined by a level of a hierarchical segmentation of a videotimeline of the video. At block 2220, in response to an inputidentifying one of the metadata segments defined by the level, a visualindication on the video timeline is updated to reflect an activeselection state of a corresponding one of the video segments defined bythe level. At block 2230, an operation is executed on the correspondingvideo segment.

Turning now to FIG. 23 , FIG. 23 illustrates a method 2300 for executinga search of extracted metadata and emphasizing matching video segmentson a video timeline, in accordance with embodiments of the presentinvention. Initially at block 2310, a hierarchical segmentation of avideo timeline of a video is accessed. The hierarchical segmentationassociates extracted metadata about the video with corresponding videosegments defined a first level of the hierarchical segmentation. Atblock 2320, an input identifying a search criterion is received. Atblock 2330, a search of the extracted metadata is executed using thesearch criterion to identify matching metadata segments of the extractedmetadata and corresponding matching video segments of video segmentsdefined by the first level of the hierarchical segmentation. At block2340, the corresponding matching video segments from the first level areemphasizing on the video timeline.

Turning now to FIG. 24 , FIG. 24 illustrates a method 2400 for executinga search of the extracted metadata and updating a selection state formatching video segments, in accordance with embodiments of the presentinvention. Initially at block 2410, a hierarchical segmentation of avideo timeline of a video is accessed. The hierarchical segmentationassociates extracted metadata about the video with corresponding videosegments defined by a level of the hierarchical segmentation. At block2420, an input identifying a keyword is received. At block 2430, asearch of the extracted metadata is executed for the keyword to identifymatching metadata segments of the extracted metadata and correspondingmatching video segments of video segments defined by the level of thehierarchical segmentation. At block 2440, a selection state for a set ofthe corresponding matching video segments from the level is updated.

Turning now to FIG. 25 , FIG. 25 illustrates a method 2500 for updatingmatching and video metadata segments in response to an input navigatingto a different level of a hierarchical segmentation, in accordance withembodiments of the present invention. In some embodiments, method 2500is performed after the steps illustrated in FIG. 23 or FIG. 24 .Initially at block 2510, in response to an input navigating from thefirst level to a different level of the hierarchical segmentation, acomposite list of the extracted metadata is segmented into an updatedset of metadata segments. The composite list segmented at locations inthe composite list corresponding to boundaries of a second set of videosegments defined by the different level of the hierarchicalsegmentation. At block 2520, an updated set of matching metadatasegments that match the search criterion is identified from the updatedset of metadata segments defined by the different level. At block 2530,an updated set of matching video segments of the second set of videosegments defined by the different level, and corresponding to theupdated set of matching metadata segments, are emphasized on the videotimeline.

Example Operating Environment

Having described an overview of embodiments of the present invention, anexample operating environment in which embodiments of the presentinvention may be implemented is described below in order to provide ageneral context for various aspects of the present invention. Referringnow to FIG. 26 in particular, an example operating environment forimplementing embodiments of the present invention is shown anddesignated generally as computing device 2600. Computing device 2600 isbut one example of a suitable computing environment and is not intendedto suggest any limitation as to the scope of use or functionality of theinvention. Neither should computing device 2600 be interpreted as havingany dependency or requirement relating to any one or combination ofcomponents illustrated.

The invention may be described in the general context of computer codeor machine-useable instructions, including computer-executableinstructions such as program modules, being executed by a computer orother machine, such as a cellular telephone, personal data assistant orother handheld device. Generally, program modules including routines,programs, objects, components, data structures, etc., refer to code thatperform particular tasks or implement particular abstract data types.The invention may be practiced in a variety of system configurations,including hand-held devices, consumer electronics, general-purposecomputers, more specialty computing devices, etc. The invention may alsobe practiced in distributed computing environments where tasks areperformed by remote-processing devices that are linked through acommunications network.

With reference to FIG. 26 , computing device 2600 includes bus 2610 thatdirectly or indirectly couples the following devices: memory 2612, oneor more processors 2614, one or more presentation components 2616,input/output (I/O) ports 2618, input/output components 2620, andillustrative power supply 2622. Bus 2610 represents what may be one ormore busses (such as an address bus, data bus, or combination thereof).Although the various blocks of FIG. 26 are shown with lines for the sakeof clarity, in reality, delineating various components is not so clear,and metaphorically, the lines would more accurately be grey and fuzzy.For example, one may consider a presentation component such as a displaydevice to be an I/O component. Also, processors have memory. Theinventor recognizes that such is the nature of the art, and reiteratesthat the diagram of FIG. 26 is merely illustrative of an examplecomputing device that can be used in connection with one or moreembodiments of the present invention. Distinction is not made betweensuch categories as “workstation,” “server,” “laptop,” “hand-helddevice,” etc., as all are contemplated within the scope of FIG. 26 andreference to “computing device.”

Computing device 2600 typically includes a variety of computer-readablemedia. Computer-readable media can be any available media that can beaccessed by computing device 2600 and includes both volatile andnonvolatile media, and removable and non-removable media. By way ofexample, and not limitation, computer-readable media may comprisecomputer storage media and communication media. Computer storage mediaincludes both volatile and nonvolatile, removable and non-removablemedia implemented in any method or technology for storage of informationsuch as computer-readable instructions, data structures, program modulesor other data. Computer storage media includes, but is not limited to,RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM,digital versatile disks (DVD) or other optical disk storage, magneticcassettes, magnetic tape, magnetic disk storage or other magneticstorage devices, or any other medium which can be used to store thedesired information and which can be accessed by computing device 2600.Computer storage media does not comprise signals per se. Communicationmedia typically embodies computer-readable instructions, datastructures, program modules or other data in a modulated data signalsuch as a carrier wave or other transport mechanism and includes anyinformation delivery media. The term “modulated data signal” means asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media includes wired media such as awired network or direct-wired connection, and wireless media such asacoustic, RF, infrared and other wireless media. Combinations of any ofthe above should also be included within the scope of computer-readablemedia.

Memory 2612 includes computer-storage media in the form of volatileand/or nonvolatile memory. The memory may be removable, non-removable,or a combination thereof. Example hardware devices include solid-statememory, hard drives, optical-disc drives, etc. Computing device 2600includes one or more processors that read data from various entitiessuch as memory 2612 or I/O components 2620. Presentation component(s)2616 present data indications to a user or other device. Examplepresentation components include a display device, speaker, printingcomponent, vibrating component, etc.

I/O ports 2618 allow computing device 2600 to be logically coupled toother devices including I/O components 2620, some of which may be builtin. Illustrative components include a microphone, joystick, game pad,satellite dish, scanner, printer, wireless device, etc. The I/Ocomponents 2620 may provide a natural user interface (NUI) thatprocesses air gestures, voice, or other physiological inputs generatedby a user. In some instances, inputs may be transmitted to anappropriate network element for further processing. An NUI may implementany combination of speech recognition, stylus recognition, facialrecognition, biometric recognition, gesture recognition both on screenand adjacent to the screen, air gestures, head and eye tracking, andtouch recognition (as described in more detail below) associated with adisplay of computing device 2600. Computing device 2600 may be equippedwith depth cameras, such as stereoscopic camera systems, infrared camerasystems, RGB camera systems, touchscreen technology, and combinations ofthese, for gesture detection and recognition. Additionally, thecomputing device 2600 may be equipped with accelerometers or gyroscopesthat enable detection of motion. The output of the accelerometers orgyroscopes may be provided to the display of computing device 2600 torender immersive augmented reality or virtual reality.

Embodiments described herein support video editing or playback. Thecomponents described herein refer to integrated components of a videoediting system. The integrated components refer to the hardwarearchitecture and software framework that support functionality using thevideo editing system. The hardware architecture refers to physicalcomponents and interrelationships thereof and the software frameworkrefers to software providing functionality that can be implemented withhardware embodied on a device.

The end-to-end software-based video editing system can operate withinthe video editing system components to operate computer hardware toprovide video editing system functionality. At a low level, hardwareprocessors execute instructions selected from a machine language (alsoreferred to as machine code or native) instruction set for a givenprocessor. The processor recognizes the native instructions and performscorresponding low level functions relating, for example, to logic,control and memory operations. Low level software written in machinecode can provide more complex functionality to higher levels ofsoftware. As used herein, computer-executable instructions includes anysoftware, including low level software written in machine code, higherlevel software such as application software and any combination thereof.In this regard, the video editing system components can manage resourcesand provide services for the video editing system functionality. Anyother variations and combinations thereof are contemplated withembodiments of the present invention.

Although some implementations are described with respect to neuralnetworks, generally embodiments may be implemented using any type ofmachine learning model(s), such as those using linear regression,logistic regression, decision trees, support vector machines (SVM),Naïve Bayes, k-nearest neighbor (Knn), K means clustering, randomforest, dimensionality reduction algorithms, gradient boostingalgorithms, neural networks (e.g., auto-encoders, convolutional,recurrent, perceptrons, Long/Short Term Memory (LSTM), Hopfield,Boltzmann, deep belief, deconvolutional, generative adversarial, liquidstate machine, etc.), and/or other types of machine learning models.

Having identified various components in the present disclosure, itshould be understood that any number of components and arrangements maybe employed to achieve the desired functionality within the scope of thepresent disclosure. For example, the components in the embodimentsdepicted in the figures are shown with lines for the sake of conceptualclarity. Other arrangements of these and other components may also beimplemented. For example, although some components are depicted assingle components, many of the elements described herein may beimplemented as discrete or distributed components or in conjunction withother components, and in any suitable combination and location. Someelements may be omitted altogether. Moreover, various functionsdescribed herein as being performed by one or more entities may becarried out by hardware, firmware, and/or software, as described below.For instance, various functions may be carried out by a processorexecuting instructions stored in memory. As such, other arrangements andelements (e.g., machines, interfaces, functions, orders, and groupingsof functions, etc.) can be used in addition to or instead of thoseshown.

The subject matter of the present invention is described withspecificity herein to meet statutory requirements. However, thedescription itself is not intended to limit the scope of this patent.Rather, the inventor has contemplated that the claimed subject mattermight also be embodied in other ways, to include different steps orcombinations of steps similar to the ones described in this document, inconjunction with other present or future technologies. Moreover,although the terms “step” and/or “block” may be used herein to connotedifferent elements of methods employed, the terms should not beinterpreted as implying any particular order among or between varioussteps herein disclosed unless and except when the order of individualsteps is explicitly described.

The present invention has been described in relation to particularembodiments, which are intended in all respects to be illustrativerather than restrictive. Alternative embodiments will become apparent tothose of ordinary skill in the art to which the present inventionpertains without departing from its scope.

From the foregoing, it will be seen that this invention is one welladapted to attain all the ends and objects set forth above, togetherwith other advantages which are obvious and inherent to the system andmethod. It will be understood that certain features and subcombinationsare of utility and may be employed without reference to other featuresand subcombinations. This is contemplated by and is within the scope ofthe claims.

What is claimed is:
 1. One or more computer storage media storingcomputer-useable instructions that, when used by one or more computingdevices, cause the one or more computing devices to perform operationscomprising: accessing a hierarchical segmentation of a video timeline ofa video, the hierarchical segmentation associating extracted metadataextracted from the video by one or more machine learning models withcorresponding video segments defined by a first level of thehierarchical segmentation; receiving, via a search bar of a userinterface, an input identifying a textual search criterion; executing asearch of the extracted metadata using the textual search criterion toidentify matching metadata segments of the extracted metadata andcorresponding matching video segments of video segments defined by thefirst level of the hierarchical segmentation; and causing the userinterface to visually emphasize, on the video timeline, thecorresponding matching video segments that are from the first level andcorrespond to the matching metadata segments that match the input in thesearch bar.
 2. The one or more computer storage media of claim 1,wherein causing the user interface to visually emphasize the matchingvideo segments comprises animating the corresponding matching videosegments on the video timeline by inducing a transient oscillatingdisplacement of representations of the corresponding matching videosegments on the video timeline.
 3. The one or more computer storagemedia of claim 1, wherein causing the user interface to visuallyemphasize the matching video segments comprises animating thecorresponding matching video segments on the video timeline by inducinga traveling wave that displaces representations of the correspondingmatching video segments on the video timeline as the traveling wavetravels down the video timeline.
 4. The one or more computer storagemedia of claim 1, the operations further comprising causing the userinterface to visually emphasize the matching metadata segments on acomposite list of the extracted metadata segmented at locations in thecomposite list corresponding to boundaries of the corresponding videosegments defined by the first level of the hierarchical segmentation. 5.The one or more computer storage media of claim 1, the operationsfurther comprising causing the user interface to visually emphasize thematching metadata segments in a metadata panel and to visually emphasizethe corresponding matching video segments on the video timeline using asame type of visual emphasis.
 6. The one or more computer storage mediaof claim 1, the operations further comprising: causing the userinterface to segment, in response to an input navigating from the firstlevel to a different level of the hierarchical segmentation, a compositelist of the extracted metadata into an updated set of metadata segments,the composite list segmented at locations in the composite listcorresponding to boundaries of a second set of video segments defined bythe different level of the hierarchical segmentation; identifying anupdated set of matching metadata segments, from the updated set ofmetadata segments defined by the different level, that match the searchcriterion; and causing the user interface to visually emphasize on thevideo timeline an updated set of matching video segments, of the secondset of video segments defined by the different level, corresponding tothe updated set of matching metadata segments.
 7. The one or morecomputer storage media of claim 1, wherein the extracted metadatacomprises transcribed audio of the corresponding video segments.
 8. Theone or more computer storage media of claim 1, wherein the extractedmetadata comprises tags visually extracted from video frames of thecorresponding video segments.
 9. The one or more computer storage mediaof claim 1, wherein the extracted metadata comprises log event tagsextracted from a temporal log associated with the video.
 10. The one ormore computer storage media of claim 1, the operations furthercomprising detecting the input selecting a metadata tag as the searchcriterion from a popup list of top metadata tags in the extractedmetadata.
 11. A computerized method comprising: accessing a hierarchicalsegmentation of a video timeline of a video, the hierarchicalsegmentation associating extracted metadata extracted from the video byone or more machine learning models with corresponding video segmentsdefined by a level of the hierarchical segmentation; receiving, via asearch bar of a user interface, an input identifying a keyword;executing a search of the extracted metadata for the keyword to identifymatching metadata segments of the extracted metadata and correspondingmatching video segments of video segments defined by the level of thehierarchical segmentation; and causing the user interface to update, onthe video timeline, a visual representation of the correspondingmatching video segments that are from the level and correspond to thematching metadata segments that match the input in the search bar. 12.The computerized method of claim 11, wherein causing the user interfaceto update the visual representation of the corresponding matching videosegments comprises animating the corresponding matching video segmentson the video timeline by inducing a transient oscillating displacementof representations of the corresponding matching video segments on thevideo timeline.
 13. The computerized method of claim 11, causing theuser interface to update the visual representation of the correspondingmatching video segments comprises animating the corresponding matchingvideo segments on the video timeline by inducing a traveling wave thatdisplaces representations of the corresponding matching video segmentson the video timeline as the traveling wave travels down the videotimeline.
 14. The computerized method of claim 11, further comprisingcausing the user interface to visually emphasize the matching metadatasegments on a composite list of the extracted metadata segmented atlocations in the composite list corresponding to boundaries of thecorresponding video segments defined by the level of the hierarchicalsegmentation.
 15. The computerized method of claim 11, furthercomprising causing, in response to an input navigating from the level toa different level of the hierarchical segmentation, the user interfaceto visually emphasize an updated set of matching video segments definedby the different level and corresponding to an updated set of matchingmetadata segments that are defined by the different level and match theinput in the search bar.
 16. The computerized method of claim 11,further comprising detecting the input identifying the keyword from apopup list of top metadata tags in the extracted metadata.
 17. Acomputer system comprising: one or more hardware processors and memoryconfigured to provide computer program instructions to the one or morehardware processors; a video interaction engine configured to use theone or more hardware processors to perform operations comprising:receiving a keyword via a search bar of a user interface; executing asearch for the keyword in extracted metadata extracted from a video byone or more machine learning models to identify matching metadatasegments of the extracted metadata and corresponding matching videosegments of video segments defined by a level of a hierarchicalsegmentation of a video timeline of the video; and causing at least oneof (i) the user interface to visually emphasize on the video timelinethe corresponding matching video segments that are from the level andcorrespond to the matching metadata segments that match the keyword inthe search bar; or (ii) the user interface to visually emphasize thematching metadata segments on a composite list of the extracted metadatasegmented at locations in the composite list corresponding to boundariesof the video segments defined by the level.
 18. The computer system ofclaim 17, wherein causing the user interface to visually emphasize thecorresponding matching video segments comprises animatingrepresentations of the corresponding matching video segments on thevideo timeline.
 19. The computer system of claim 17, the operationsfurther comprising updating the matching metadata segments and thecorresponding matching video segments, in response to an inputnavigating from the level to a different level of the hierarchicalsegmentation, using a second set of boundaries defined by the differentlevel.
 20. The computer system of claim 17, the operations furthercomprising: placing a set of the corresponding matching video segmentsfrom the level into an operational queue; and executing an operation ona set of the video segments in the operational queue.